[
https://issues.apache.org/jira/browse/CALCITE-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114302#comment-17114302
]
Haisheng Yuan commented on CALCITE-3221:
----------------------------------------
Looks like CALCITE-4017 has a dependency on this JIRA.
I would not suggest adding a new physical operator {{EnumerableMergeUnion}}. We
can extends {{EnumerableUnion}} by adding a field {{collation}}, to indicate
whether to preserve ordering and what is the order. Default is EMPTY, which
means no ordering needs to be preserved. Because if the parent operator doesn't
require any collation, it is always good to not preserve order. If it is
required by parent operator, we can pass down the required collation to UNION's
children on demand, and returns its new collation, by overriding
{{EnumerableUnion#passThroughTraits()}}.
> Add a sort-merge union algorithm
> --------------------------------
>
> Key: CALCITE-3221
> URL: https://issues.apache.org/jira/browse/CALCITE-3221
> Project: Calcite
> Issue Type: Improvement
> Components: core
> Affects Versions: 1.19.0
> Reporter: Stamatis Zampetakis
> Priority: Minor
>
> Currently, the union operation offered by Calcite is based on a {{HashSet}}
> (see
> [EnumerableDefaults.union|https://github.com/apache/calcite/blob/d98856bf1a5f5c151d004b769e14bdd368a67234/linq4j/src/main/java/org/apache/calcite/linq4j/EnumerableDefaults.java#L2747])
> and necessitates reading in memory all rows before returning a single
> result.
> Apart from increased memory consumption the operator is blocking and also
> destroys the order of its inputs.
> The goal of this issue is to add a new union algorithm (EnumerableMergeUnion
> ?) exploiting the fact that the inputs are sorted which consumes less memory
> and retains the order of its inputs.
> Most likely the implementation of the merge join can be useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
