kaisun2000 commented on issue #12721: URL: https://github.com/apache/druid/issues/12721#issuecomment-1984935814
Copy the [slack channel comment](https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1709862138570539?thread_ts=1704511189.243569&cid=C030CMF6B70) here: For the past two month, I worked on some query performance tuning in our system, mainly to reduce the query latency and specifically in the real time path. In the mean time, I find that a query insight tool like proposed above would be very valuable. More specifically, the following would be useful and some of them are missing: 1/ per segment query processing time 2/ per segment waiting time for the processing queue and current queue size 3/ per hydrant query processing time -- missing 4/ per hydrant waiting time thread time -- if paralleled, missing 5/ per query merge buffer acquisition time and current waiting queue size -- missing 6/ post segment processing grouper potentially spilling to disk time -- missing These information should not be limited to only brokers, but also data node (peons and historicals). The main idea is attribute time spent for each query in each stage and report contention points statistics such as the thread pool queue length while waiting, or merge buffer queue length while waiting, or spilling to disk activity for groupers. report this data collectively to some UI so that admin can have a direct insights above queries and why they may be slow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
