egor-ryashin commented on a change in pull request #7088: Improve parallelism
of zookeeper based segment change processing
URL: https://github.com/apache/incubator-druid/pull/7088#discussion_r266250834
##########
File path: docs/content/configuration/index.md
##########
@@ -1251,7 +1251,11 @@ These Historical configurations can be defined in the
`historical/runtime.proper
|`druid.segmentCache.infoDir`|Historical nodes keep track of the segments they
are serving so that when the process is restarted they can reload the same
segments without waiting for the Coordinator to reassign. This path defines
where this metadata is kept. Directory will be created if
needed.|${first_location}/info_dir|
|`druid.segmentCache.announceIntervalMillis`|How frequently to announce
segments while segments are loading from cache. Set this value to zero to wait
for all segments to be loaded before announcing.|5000 (5 seconds)|
|`druid.segmentCache.numLoadingThreads`|How many segments to drop or load
concurrently from from deep storage.|10|
-|`druid.segmentCache.numBootstrapThreads`|How many segments to load
concurrently from local storage at startup.|Same as numLoadingThreads|
+|`druid.coordinator.loadqueuepeon.curator.numCreateThreads`|Number of threads
creating zk nodes corresponding to segments that need to be loaded or
dropped.|10|
+|`druid.coordinator.loadqueuepeon.curator.numCallbackThreads`|Number of
threads for executing callback actions associated with loading or dropping of
segments.|2|
+|`druid.coordinator.loadqueuepeon.curator.numMonitorThreads`|Number of threads
to use for monitoring deletion of zk nodes|1|
+|`druid.coordinator.curator.create.zknode.batchSize`|Number of zk nodes to
create in one iteration.|5000|
+|`druid.coordinator.curator.create.zknode.repeatDelay`|Delay before creating
next batch of zk nodes|PT1M|
Review comment:
I wonder how should the operator calculate the delay and batch size? I think
it's complex right now.
For example I understand that
`delay*batch_size*avg_segment_size<=node_network_throughput`
But I can't figure out what number to select here, as they depend on each
other. Probably there should be one parameter like
`max_node_network_throughput` and the Peon keeps sending load request until the
limit is hit, then the Peon delays requests until the quota is replenished.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
