FrankChen021 commented on PR #20183: URL: https://github.com/apache/druid/pull/20183#issuecomment-5595243757
> Thanks for these changes @FrankChen021! The `sys.*` tables have historically been slow and less performant, especially `sys.segments` and `sys.tasks`. > > I wonder if we can benchmark `sys.tasks` with thousands of tasks to see how these changes perform at scale with the native support + filter pushdown. > > Also, regarding the scope of these changes, I feel it would be helpful to break this into a few more patches for ease of review. Perhaps something like: > > * Separate PRs for `sys.tasks` and `sys.server_properties`, isolating the appropriate wirings, filter pushdown mechanisms and tests to make it work for each table. > * Handle `DECOUPLED` planner support in a separate change, since `COUPLED` is the default planner strategy today (and `DECOUPLED` is currently undocumented). Thanks for reviewing. I deliberately chose the sys.tasks and sys.server_properties included in this PR to demonstrate how the native query for system tables are supported. These two system tables has different fan-out path and push down policies: 1. for sys.server_properties, we need to fan out sub queries to all nodes, while for sys.tasks we need to fan out the query to leader overlord only. This is also the header `X-Druid-Native-Query-Route` is introduced to server the purpose 2. for sys.tasks, it also demonstrate how filters are pushed down, and demonstrate the the other most valueable gain we want to achieve. If we move any of these system table out of this PR, we can't have a full picture of the change, and understand how the small framework for the system tables work. As for DECOUPLED mode, only a few files are involved, I think it's better to included in this PR. To address your concern, I can split the changes for sys.tasks out of this one. But We still need to have a full picture of the core change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
