gianm commented on issue #19456: URL: https://github.com/apache/druid/issues/19456#issuecomment-4485598377
Interesting stuff. Will you be working on this? One nudge, you might have luck doing things in the MSQ path (the `FrameProcessor`s) by using a `RowsAndColumns` backed by Arrow rather than Druid's builtin `Frame`. It may be easier to experiment with the nonleaf paths first. The nonleaf paths are the ones that read from `ReadableFrameChannel`. They today read `Frame` and could probably be swapped out for something that processes Arrow without too much fuss. The leaf paths— the ones that read `Segment`— may require some additional care to avoid materializing things too early. Druid's leaf computation pipeline pushes down a lot of operations such that they run directly on a `Cursor` or `VectorCursor`. Even a fairly complex set of operations like filter -> project -> broadcast hash join -> aggregate can all get pushed down, with a `Frame` being created with the results of the aggregation (in this example). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
