lokesh-lingarajan opened a new issue #12318: URL: https://github.com/apache/druid/issues/12318
We are currently running a large number of tasks (~1k) in our production and we have started seeing metadata store network throughput increase by 3x as we fetch tasks in UI console and hence degrading performance. This is because the data model for druid_tasks does not have task status as an individual column instead its embedded inside the payload. Because of this model, UI fetches all of the tasks, parses them and then filters out in the widgets. Faster approach to this solution is in UI, where we move towards overlord in fetching the task listing and having sql as fallback. A more longer approach is to solve the data model in the metadata store and let the filter happen at the metadata store itself. The second solution is more involved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
