leventov commented on a change in pull request #7088: Improve parallelism of
zookeeper based segment change processing
URL: https://github.com/apache/incubator-druid/pull/7088#discussion_r277350868
##########
File path: docs/content/configuration/index.md
##########
@@ -1254,9 +1254,9 @@ These Historical configurations can be defined in the
`historical/runtime.proper
|`druid.segmentCache.dropSegmentDelayMillis`|How long a process delays before
completely dropping segment.|30000 (30 seconds)|
|`druid.segmentCache.infoDir`|Historical processes keep track of the segments
they are serving so that when the process is restarted they can reload the same
segments without waiting for the Coordinator to reassign. This path defines
where this metadata is kept. Directory will be created if
needed.|${first_location}/info_dir|
|`druid.segmentCache.announceIntervalMillis`|How frequently to announce
segments while segments are loading from cache. Set this value to zero to wait
for all segments to be loaded before announcing.|5000 (5 seconds)|
-|`druid.segmentCache.numLoadingThreads`|How many segments to drop or load
concurrently from from deep storage.|10|
-|`druid.coordinator.loadqueuepeon.curator.numCallbackThreads`|Number of
threads for executing callback actions associated with loading or dropping of
segments.|2|
-|`druid.coordinator.loadqueuepeon.curator.numMonitorThreads`|Number of threads
to use for monitoring deletion of zk nodes|1|
+|`druid.segmentCache.numLoadingThreads`|How many segments to drop or load
concurrently from deep storage. Note that the work of loading segments involves
downloading segments from deep storage, decompressing them and loading them to
a memory mapped location. So the work is not all I/O Bound. Depending on CPU
and network load, one could possibly increase this config to a higher
value.|Number of cores|
+|`druid.coordinator.loadqueuepeon.curator.numCallbackThreads`|Number of
threads for executing callback actions associated with loading or dropping of
segments. One might want to increase this number when noticing clusters are
lagging behind w.r.t. balancing segments across historical nodes.|2|
+|`druid.coordinator.loadqueuepeon.curator.numMonitorThreads`|Number of threads
to use for monitoring deletion of zk nodes. Tasks in this pool get scheduled to
run at time `druid.coordinator.load.timeout` after a segment is added to the
queue. Increase this number if you see segments not getting loaded or dropped
even after `druid.coordinator.load.timeout` since it is possible they are not
getting re-assigned to the queue of other historicals soon enough.|1|
Review comment:
The reassignment is done from within the thread pool? I think that can only
be done in during a Coordinator run (from `DruidCoordinator.exec`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
