acherla commented on issue #13692: URL: https://github.com/apache/druid/issues/13692#issuecomment-1490173435
Re-opening this just to provide updates to folks who are running large druid clusters. This thread might help them So we ran a few performance tests here based on 25.0.0 and we are finding some interesting results. 1. segment handoff time did improve, however enabling chatAsync lead to the Overlord getting flooded with requests that would result in thousands of task failures. We tested this under extreme load (close to 3K Peons) running just Kafka Indexing jobs, it seems likely due to the number of chat requests that are flooding the overlord with requests and the overlord would completely stall out (CPU usage would spike to 80-90% or higher) to the point where it was inaccessible from the management console. We are currently exploring two options: 1. Federating the coordinator/overlord to separate machines, right now we are running the active/backup as a coordinator/overlord combo. We are planning to federate out the overlord and coordinator going forward to separate machines 2. setting druid.indexer.server.maxChatRequests to 1 to throttle the chattiness of peons to the overlord (does this potentially have negative consequences? IE longer handoff times?) 3. Setting chatAsync to false, we noticed a huge number of task failures due to the overlord throwing jetty connection refused exceptions when it was enabled potentially due to the number of chat requests flooding the overlord due to the number of peons we are running. @AmatyaAvadhanula any other tips you might be able to recommend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
