jihoonson commented on issue #6171: hetrogenous peon configurations URL: https://github.com/apache/incubator-druid/issues/6171#issuecomment-412720249 The exact number matters. I would say thousands of queries shouldn't have performance impact. Let's say a single query takes 10 ms (the number came from one of our customers). If you have 10,000 queries to execute in 1 min, you need to execute about 1.7 queries per sec which can be easily handled. > this should still have the problem of 1 large datasource requiring settting up tons of tiny replicas which can be wasteful and tough to manage, compared to simply specifying a jvm parameter in the supervisor config Why is it wasteful? The only thing you need to do is to set the number of replicas in your supervisorSpec. Allowing heterogeneous resource configurations for peon _does_ make Druid's resource management complicated. It causes new issues like how to allocate tasks to MMs to maximize the resource utilization. > you already allow setting number of worker threads in supervisor spec. since direct memory is allocated proportional to worker threads, you are exposing half configuration which will inevitably result in crashes. Do you mean `workerThreads` [here](http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html#kafkasupervisortuningconfig)? As documented, it's `The number of threads that will be used by the supervisor for asynchronous operations`, not the number of query processing threads.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
