gianm opened a new pull request, #12848:
URL: https://github.com/apache/druid/pull/12848
Follow-up to #12745. This patch adds three new concepts:
1) Frame channels are interfaces for doing nonblocking reads and writes
of frames.
2) Frame processors are interfaces for doing nonblocking processing of
frames received from input channels and sent to output channels.
3) Cluster-by keys, which can be used for sorting or partitioning.
The patch also adds SuperSorter, a user of these concepts, both to
illustrate how they are used, and also because it is going to be useful
in future work.
Central classes:
-
[ReadableFrameChannel](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/channel/ReadableFrameChannel.java),
nonblocking source of frames. Implementations include
BlockingQueueFrameChannel (in-memory channel that implements both interfaces),
ReadableFileFrameChannel (file-based channel), ReadableByteChunksFrameChannel
(byte-stream-based channel), and others.
-
[WritableFrameChannel](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/channel/WritableFrameChannel.java),
nonblocking sink for frames. Implementations include BlockingQueueFrameChannel
and WritableStreamFrameChannel (byte-stream-based channel).
-
[ClusterBy](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/key/ClusterBy.java),
a sorting or partitioning key.
-
[FrameProcessor](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/processor/FrameProcessor.java),
nonblocking processor of frames. Implementations include FrameChannelBatcher,
FrameChannelMerger, and FrameChannelMuxer.
-
[FrameProcessorExecutor](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/processor/FrameProcessorExecutor.java),
an executor service that runs FrameProcessors.
-
[SuperSorter](https://github.com/gianm/druid/blob/frame-processor/processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java),
a class that uses frame channels and processors to do parallel external merge
sort of any amount of data (as long as there is enough disk space).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
