[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-551243337 Thanks for the review everyone! This is an automated message from

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-549966959 >@clintropolis thanks for all the benchmarks, I haven't had the opportunity to look at the new developments yet but get back to

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-54976 ### simulated heavy load I collected another round of data using the same benchmarks as my 'more realistic worst case' comment,

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-549253205 ### more realistic worst case I reworked the JMH thread based benchmark to use thread groups to examine what happens in a more

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548611095 >Thank you for the detailed benchmark results! It looks great but I wonder how the default configuration works under more

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548155872 ### my conclusions I think the performance degradation under load spikes falls within what I consider acceptable range.

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548152735 ### worst case The other really important part to highlight, is to determine how this approach degrades under increasing

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548142920 ### best case All benchmarks for this phase were done on an AWS `m5.8xl` which has 16 physical cores, 32 hyper-threads. The

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548141222 ### Hyper-threading and default ForkJoinPool parallelism and per query parallelism configuration values Another side-note

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548140327 ### ParallelMergeCombiningSequenceBenchmark and ParallelMergeCombiningSequenceJmhThreadsBenchmark These are the new benchmarks

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-548137628 Ok, apologies for going dark while I was running benchmarks and making minor adjustments over the last couple weeks; I have some

[GitHub] [incubator-druid] clintropolis commented on issue #8578: parallel broker merges on fork join pool

clintropolis commented on issue #8578: parallel broker merges on fork join pool URL: https://github.com/apache/incubator-druid/pull/8578#issuecomment-537802979 >I'm strongly in favor of making the new behavior opt-out rather than opt-in, and removing the old behavior after a few versions