[
https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902548#comment-16902548
]
sridhar Reddy commented on BEAM-7049:
-------------------------------------
[~amaliujia] Thanks for checking in with me.
I just finished a couple of tests and can you please confirm if I am doing the
right tests
created three PCollections
---------------
PCollection<Row> outputStreamOne =
inputTable.apply(SqlTransform.query("select c1, c2, c3 from PCOLLECTION "));
PCollection<Row> outputStreamTwo =
inputTable2.apply(SqlTransform.query("select c1, c2, c3 from PCOLLECTION "));
PCollection<Row> outputStreamThree =
inputTable3.apply(SqlTransform.query("select c1, c2, c3 from PCOLLECTION "));
--------
used union on 3 Pcollections
----------
PCollection<Row> outputStreamResult =
PCollectionTuple.of(new TupleTag<>("FirstOne"), outputStreamOne)
.and(new TupleTag<>("SecondOne"), outputStreamTwo)
.and(new TupleTag<>("ThirdOne"), outputStreamThree)
.apply(SqlTransform.query("select c2 from SecondOne union " +
"select c2 from FirstOne union select c2 from ThirdOne" ));
-----------------
It seems to be working fine. If I used a simple union that seems to be working
fine too
-----------------------
PCollection<Row> outputStreamResult =
PCollectionTuple.of(new TupleTag<>("FirstOne"), outputStreamOne)
.and(new TupleTag<>("SecondOne"), outputStreamTwo)
.and(new TupleTag<>("ThirdOne"), outputStreamThree)
.apply(SqlTransform.query("select 1 from SecondOne union " +
"select 2 from FirstOne union select 3 from ThirdOne" ));
-----------------
Please let me know your thoughts.
> BeamUnionRel should work on mutiple input
> ------------------------------------------
>
> Key: BEAM-7049
> URL: https://issues.apache.org/jira/browse/BEAM-7049
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Rui Wang
> Assignee: sridhar Reddy
> Priority: Major
>
> BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c`
> will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If
> BeamUnionRel can handle multiple shuffles, we will have only one shuffle
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
