gianm commented on issue #19456: URL: https://github.com/apache/druid/issues/19456#issuecomment-4486406707
> Another thing I want to call out here is starting to move towards deprecating and unifying the available engines in Druid. Currently I see a lot of work being done on various engines (dart, msq, native, etc.) and what I think we should be aiming towards is the native processing core to be built out on whatever the decided "next" gen engine is (and its operator interfaces). This is another reason I mentioned a nudge towards MSQ. In addition to having a Frame concept that should be relatively straightforward to swap out for an Arrow batch, it is also a generally more capable engine and one that I expect we'll standardize on in time. Fwiw, MSQ and Dart are the same underlying engine. Really MSQ is the engine and Dart is a "profile" of the engine. It is a mode where the Broker runs query controllers and the Historicals are query workers. The other MSQ profile is Tasks, where the controllers and workers are all launched as tasks. Anyway, I think a lot of interesting experiments could be done without necessarily needing to get into the leaf processing stuff. For example, the MSQ `sortMergeJoin` implementation operates purely on Frames and could be swapped out for something else. The MSQ `groupBy` implementation has a per-segment partial-aggregation phase that emits Frames, followed by a phase that sorts those Frames, followed by a merge of sorted streams of Frames. Another experiment there could keep the per-segment partial-aggregation stuff as-is, and swap out the sort-to-sort-merge part with some other form of aggregation operation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
