gianm opened a new pull request, #18909: URL: https://github.com/apache/druid/pull/18909
This patch introduces a "WireTransferable" interface that is used to serialize batches of results represented by RowsAndColumns objects. This allows query implementations to control the exact manner in which their result batches are modeled, serialized, and deserialized, which enables greater efficiency. For example, this would allow an MSQ implementation of TopN (which does not currently exist) to use a data structure between leaf and merge stages that matches what it uses in its native implementation. It would also allow this data structure to be serialized efficiently, without needing to use JSON/Smile. WireTransferable is used in two places: 1) In ObjectMappers, when they read or write RowsAndColumns objects. 2) In MSQ channels, when frames or other batches are read or written to disk or network. Prior to this patch, the ObjectMapper path only handled FrameRowsAndColumns, and the channel path only handled Frames, in both cases using a Frame-specific serialization format. These Frame-specific serialization formats are still used by default. The new FrameWireTransferable format can be enabled in both places by setting `druid.serde.rac.useLegacyFrameSerialization = false`. As part of this change, the ReadableFrameChannels and WritableFrameChannels used by MSQ are generalized to allow transmitting any RowsAndColumns, not just Frames. When Frames are transmitted, they are enclosed in a thin FrameRowsAndColumns wrapper. In the interest of minimizing the size of the diff, the names of the classes are not changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
