clintropolis edited a comment on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-549966959 >@clintropolis thanks for all the benchmarks, I haven't had the opportunity to look at the new developments yet but get back to reviewing this week. Yeah, no problem, thanks for asking the hard questions to make me collect them, the result is the PR is in a better state than before them :metal:. The production part of the code hasn't really changed much in the last couple of weeks other than a few lines to change behavior of the parallelism computing method, and mostly changes to the default values. >Given the sensitivity of performance to availableProcessors() returned value, it might be good to make that area a bit configurable if not already. I will hopefully offer more specific suggestion when reviewing again. This stuff should all be controllable via configs, `druid.processing.merge.pool.parallelism` to control the FJP pool size, and `druid.merge.pool.defaultMaxQueryParallelism` to control individual query max parallelism (this one can also be set on the query context through `parallelMergeParallelism`). In a follow-up I think I will also try to add additional information to the cluster tuning guide docs, since I have a pretty good idea how this implementation performs now which I think we can use to advise operators.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
