[
https://issues.apache.org/jira/browse/BEAM-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274747#comment-16274747
]
ASF GitHub Bot commented on BEAM-3171:
--------------------------------------
XuMingmin commented on a change in pull request #4196: [BEAM-3171] convert a
join into lookup
URL: https://github.com/apache/beam/pull/4196#discussion_r154414525
##########
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
##########
@@ -98,9 +101,14 @@ public BeamJoinRel(RelOptCluster cluster, RelTraitSet
traits, RelNode left, RelN
throws Exception {
BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
BeamRecordSqlType leftRowType =
CalciteUtils.toBeamRowType(left.getRowType());
- PCollection<BeamRecord> leftRows =
leftRelNode.buildBeamPipeline(inputPCollections, sqlEnv);
-
final BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
+
+ if (!seekable(leftRelNode, sqlEnv) && seekable(rightRelNode, sqlEnv)) {
Review comment:
Technically yes, but this is a special optiomization rule and I don't want
to have too much flexibility. In our development guide it's restricted to be
`FACT_TABLE JOIN LOOKUP_TABLE` to make it easy-to-understand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> convert a join into lookup
> --------------------------
>
> Key: BEAM-3171
> URL: https://issues.apache.org/jira/browse/BEAM-3171
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
> Reporter: Xu Mingmin
> Assignee: Xu Mingmin
> Labels: experimental
>
> We use BeamSQL to run streaming jobs mostly, and add a join_as_lookup
> improvement(internal branch) to cover the streaming-to-batch case(similar as
> [1]). I could submit a PR as experimental if people are interested.
> The rough solution is, if one source of join node implements
> {{BeamSeekableTable}} and the other is not, then the join node is converted
> to a fact-lookup operation.
> Ref:
> [1]
> https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A/edit?usp=sharing
>
> [~xumingming] [~takidau] for any comments
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
