imply-cheddar opened a new pull request, #13694: URL: https://github.com/apache/druid/pull/13694
This enables "merge" style operations that combine multiple streams. This change includes a naive implementation of one such merge operator just to provide concrete evidence that the refactoring is effective. The primary intent of the change can be seen by looking at the `Operator` interface. The change amounts to the addition of a `Signal` enumeration that allows the Receiver to signal back to the Operator that it needs to pause things. This then returns control to the caller to do something else (like call another Operator). The vast majority of operators should never have to deal with this, but some that do merges will. For a view into something that does a merge, this PR also includes a `SortedInnerJoinOperator` which implements a multi-way inner join across operators. The class is almost entirely business logic for conducting the join rather than interactions with the Operator interface, so it is quite large, but the intent was to prove that it is possible to create a meaningful "merging" operator on top of the interface changes, which it achieves. I would recommend starting with the `SortedInnerJoinOperatorTest` when reviewing just because that will show the intended usage of the operator. Note, however, that this operator was created merely to exercise the interface change, as such it is not wired up into planning no is there a query path that can exercise it yet. Now that I am certain the interface can handle all of the needs of the data processing pipeline, I intend to go back to fleshing out the full test suite for window functions and then can come back to this. In terms of validation of the interface, there is likely one more validation to conduct, though I am fairly certain that it will be relatively simple and leave as a future exercise: the implementation of a `FrameProcessor` based on `Operator`. This PR has: - [x] been self-reviewed. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
