kevinrr888 opened a new pull request, #5301:
URL: https://github.com/apache/accumulo/pull/5301
This PR:
- Changes FATE from having one pool of workers per FATE type (META and USER)
to having a set of pools per FATE type which each work on their own subset of
FATE operations:
- This is accomplished through a new class `FateExecutor`. Each FATE
has a set of `FateExecutor`s. Each `FateExecutor` is assigned to one set of
FATE operations, has a `WorkFinder` thread, and a pool of `TransactionRunner`
threads. The `WorkFinder` finds transactions this `FateExecutor` can work on,
and assigns them to the `TransactionRunner`s which work on them.
- `Fate` class now only handles the `DeadReservationCleaner` thread and
the `FatePoolsWatcher` thread (previously `FatePoolWatcher`). The
`DeadReservationCleaner` is unchanged. `FatePoolsWatcher` now oversees all the
`FateExecutor`s/pools. The thread periodically checks for changes to the
config, resizing any pools as necessary, safely stopping (if a thread is
working on a transaction, it is not terminated until it finishes that work) any
`FateExecutor`s that are invalidated by config changes (i.e., its assigned set
of FATE operations is no longer present in the config), and starting any
new/replacement `FateExecutor`s needed. The `FatePoolsWatcher` also keeps track
of the number of idle threads to warn the user if a pool is experiencing a high
load and should have its pool size increased (this functionality is unchanged
from before, only difference is now a slightly different message is logged when
warning the user).
- The set of pools is determined by the new
`MANAGER_USER_FATE_CONFIG`/`MANAGER_META_FATE_CONFIG` properties (replaces
`MANAGER_FATE_THREADPOOL_SIZE`).
- Another difference from this change is now USER and META FATEs can be
configured differently whereas previously they shared the same property
(`MANAGER_FATE_THREADPOOL_SIZE`)
- These properties are json where the keys are a set of FATE operations
and values are a pool size.
- The properties are ensured to be valid json, pool size > 0, contain
all FATE operations, contain no invalid FATE operations, and have no duplicate
FATE operations
- The FATE operations can be split across any number of pools of any
size.
- Tests which used the `MANAGER_FATE_THREADPOOL_SIZE` have now been changed
to `MANAGER_USER_FATE_CONFIG`/`MANAGER_META_FATE_CONFIG` properties. To keep
testing functionality the same, the properties set are json of 1 key/value
where the key is all FATE ops and the value is the value originally used in the
test.
- `FatePoolResizeIT`/`MetaFatePoolResizeIT`/`UserFatePoolResizeIT` have been
renamed to
`FatePoolsWatcherIT`/`MetaFatePoolsWatcherIT`/`UserFatePoolsWatcherIT` since
they now test other functionality of the `FatePoolsWatcher` not just pool
resizing. The tests have also been completely changed since resizing has
completely changed.
I also ensured all FATE tests still pass and sunny day still passes. I have
not yet run the full set of ITs for this.
closes #5130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
