paul-rogers commented on issue #12262: URL: https://github.com/apache/druid/issues/12262#issuecomment-1183481179
@gianm, sounds like a great set of improvements! The way we achieved these goals in other systems (Drill, Presto, Impala, Hive, etc.) is to use the classic optimizer + operator DAG architecture. The optimizer allows queries to handle a wide variety of use cases (anything that can be expressed in SQL): scans, groupings, joins, windowing, grouping over joins over scans topped by windowing, etc. The optimizer (AKA "planner") handles distribution: whether that be classic Druid scatter-gather or a more general multi-stage plan. The optimizer also handles tasks such as data locality, resource balancing, etc. This is how Drill, Impala and Presto handle those monster 1000 like SQL statements will optimal performance (given the data format they work with.) Druid may never need to handle complex BI queries. Even in our domain, the occasional shuffle join, cold-tier query, union, wide merge and other operations would benefit from the optimizer/operator architecture. While, for Druid, these are stretch goals, for those other engines they are warm-up exercises because the architecture is designed for flexibility. The reason for the optimizer/operator architecture is enable the software (the planner) to make decisions about query execution, resources, distribution and so on. Query engines do that so the user doesn't have to. For example, with Druid, we need experts such as yourself to optimize queries and the cluster. But, most other products, the software itself does the query optimization work via the optimizer. Both approaches work, of course. Many products see automation as a win to lower costs, promote ease-of-use, handle a range of use cases, etc. The usual argument about automation vs. manual tasks. The planner needs a way to execute, which is typically a set of operators that the planner can rearrange as needed to produce any given query. Think of Calcite with its built-in LinqJ solution. Drill, Impala and Presto all are based on "lego block" operators which the optimizer arranges as needed for a given query. In the MSQ work, frame processors can be seen as heavy-weight operators: they can be composed, but not quite with the flexibility and nuance of lighter-weight operators. Operators need a way to process data. Frames seem like a great solution. DAGs need a way to shuffle data. Both HTTP and more sophisticated solutions can work and are isolated in the "exchange" operators. Druid, of course, has classically used its ad-hoc query runner/sequence/native query stack which has been great for a decade, but, of course, is optimized (and limited to) the scatter-gather architecture. It can be extended with new native queries, more ad-hoc logic, and so on. This extension, however, cannot address the fundamental limitation: that every SQL query is mapped into a set of one or more native queries, and the Broker combines the results. That's the pattern we want to generalize if we want to expand use cases. The good news is that that classic architecture does, obviously, have the key data pipeline operations, just wrapped in query runner/sequence code. Some prototypes were done to show how we could evolve this into the optimizer/operator architecture. See [this issue](https://github.com/apache/druid/issues/11933). The "cache-aware work assignment" mentioned above is one part of what an optimizer normally does, and would make a great contribution toward an optimizer. So, a good question for the team to consider is if we can get where we want by continuing to roll our own approach, extended with the features outlined above (frames, better exchanges), or if we actually need the flexibility of the industry-standard optimizer/operator architecture. If we do, there is a path to get there, based on the team's excellent work to date, including the mechanisms discussed above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
