[
https://issues.apache.org/jira/browse/PHOENIX-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387622#comment-14387622
]
James Taylor commented on PHOENIX-1580:
---------------------------------------
Thanks for the revisions, [~ayingshu]. Here's some feedback:
- You have to push down the ORDER BY and LIMIT to the individual scans in order
for UNION ALL to scale. It's very important to understand conceptually what is
going on here. When you have a set of iterators that are already in sorted
order, you can do a merge sort between them. Merge sorts are good because they
don't require much memory and scale well for big data. If you had to order all
the rows from all the scans on the client, you'd essentially need to buffer all
of them on the client which is inherently a bad idea. The key here is how you
combine the set of individual iterators. Instead of using a
ConcatResultIterator, which just concatenates the scan results together, you
need to do a merge sort between them using the MergeSortTopNResultIterator.
Something like this:
{code}
+ public final ResultIterator iterator(final List<? extends SQLCloseable>
dependencies) throws SQLException {
+ ResultIterator scanner;
+
+ List<PeekingResultIterator> pIterators = new
ArrayList<PeekingResultIterator>();
+ for (QueryPlan plan : this.getPlans()) {
+ pIterators.add(LookAheadResultIterator.wrap(plan.iterator()));
+ }
+
+ if (!orderBy.getOrderByExpressions().isEmpty()) { // TopN
+ scanner = new MergeSortTopNResultIterator(pIterators, limit,
orderBy.getOrderByExpressions());
+ } else {
+ scanner = new ConcatResultIterator(pIterators);
+ if (limit != null) {
+ scanner = new LimitingResultIterator(scanner, limit);
+ }
+ }
+
+ return scanner;
+ }
+
{code}
- Don't make the WildcardParseNode constructor public, but instead go through
the static factory method.
{code}
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
b/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
index 9922c3f..ddf6f3b 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/parse/WildcardParseNode.java
@@ -37,7 +37,7 @@ public class WildcardParseNode extends TerminalParseNode {
private final boolean isRewrite;
- private WildcardParseNode(boolean isRewrite) {
+ public WildcardParseNode(boolean isRewrite) {
this.isRewrite = isRewrite;
}
{code}
- Similarly with ConcatResultIterator, don't make its constructor public,
instead go through the static factory method:
{code}
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
b/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
index fcc88aa..b3cf83b 100644
---
a/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
+++
b/phoenix-core/src/main/java/org/apache/phoenix/iterate/ConcatResultIterator.java
@@ -40,7 +40,7 @@ public class ConcatResultIterator implements
PeekingResultIterator {
this.resultIterators = iterators;
}
- private ConcatResultIterator(List<PeekingResultIterator> iterators) {
+ public ConcatResultIterator(List<PeekingResultIterator> iterators) {
this.resultIterators = null;
this.iterators = iterators;
}
{code}
> Support UNION ALL
> -----------------
>
> Key: PHOENIX-1580
> URL: https://issues.apache.org/jira/browse/PHOENIX-1580
> Project: Phoenix
> Issue Type: Bug
> Reporter: Alicia Ying Shu
> Assignee: Alicia Ying Shu
> Attachments: PHOENIX-1580-grammar.patch, Phoenix-1580-v1.patch,
> phoenix-1580-v1-wipe.patch, phoenix-1580.patch, unionall-wipe.patch
>
>
> Select * from T1
> UNION ALL
> Select * from T2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
