twalthr commented on code in PR #26230:
URL: https://github.com/apache/flink/pull/26230#discussion_r1975476456
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/IntervalJoinRestoreTest.java:
##########
@@ -35,7 +35,13 @@ public IntervalJoinRestoreTest() {
public List<TableTestProgram> programs() {
return Arrays.asList(
IntervalJoinTestPrograms.INTERVAL_JOIN_EVENT_TIME,
+
IntervalJoinTestPrograms.INTERVAL_JOIN_EVENT_TIME_UPDATING_SOURCE,
Review Comment:
We don't need a restore test. Restoring doesn't need to be tested.
I added a `SemanticTestBase` recently to test an execution only:
Take this as an example: https://github.com/apache/flink/pull/26236
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala:
##########
@@ -287,6 +287,39 @@ class FlinkChangelogModeInferenceProgram extends
FlinkOptimizeProgram[StreamOpti
val leftTrait =
children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
createNewNode(temporalJoin, children, leftTrait, requiredTrait,
requester)
+ case intervalJoin: StreamPhysicalIntervalJoin =>
+ // IntervalJoin only support consuming insert-only and producing
insert-only changes
+ // However, when the input contains updates convert to RegularJoin
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet =
ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ val containsUpdatesOrDeletes = inputModifyKindSet.contains(
+ ModifyKind.UPDATE) || inputModifyKindSet.contains(ModifyKind.DELETE)
+ if (containsUpdatesOrDeletes) {
+ // Convert to regular join if the input contains Updates
+ val isInnerJoin = intervalJoin.joinSpec.getJoinType ==
FlinkJoinType.INNER
Review Comment:
why only inner and not all?
##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/optimize/program/FlinkChangelogModeInferenceProgram.scala:
##########
@@ -287,6 +287,39 @@ class FlinkChangelogModeInferenceProgram extends
FlinkOptimizeProgram[StreamOpti
val leftTrait =
children.head.getTraitSet.getTrait(ModifyKindSetTraitDef.INSTANCE)
createNewNode(temporalJoin, children, leftTrait, requiredTrait,
requester)
+ case intervalJoin: StreamPhysicalIntervalJoin =>
+ // IntervalJoin only support consuming insert-only and producing
insert-only changes
+ // However, when the input contains updates convert to RegularJoin
+ val children = visitChildren(rel, ModifyKindSetTrait.ALL_CHANGES)
+ val inputModifyKindSet =
ModifyKindSet.union(children.map(getModifyKindSet): _*)
+ val containsUpdatesOrDeletes = inputModifyKindSet.contains(
+ ModifyKind.UPDATE) || inputModifyKindSet.contains(ModifyKind.DELETE)
+ if (containsUpdatesOrDeletes) {
+ // Convert to regular join if the input contains Updates
+ val isInnerJoin = intervalJoin.joinSpec.getJoinType ==
FlinkJoinType.INNER
+ val providedTrait = if (isInnerJoin) {
+ // forward left and right modify operations
+ val leftKindSet = getModifyKindSet(children.head)
+ val rightKindSet = getModifyKindSet(children.last)
+ new ModifyKindSetTrait(leftKindSet.union(rightKindSet))
+ } else {
+ // otherwise, it may produce any kinds of changes
+ ModifyKindSetTrait.ALL_CHANGES
+ }
+ val regularJoin = new StreamPhysicalJoin(
+ rel.getCluster,
+ rel.getTraitSet,
+ intervalJoin.getLeft,
+ intervalJoin.getRight,
+ intervalJoin.getCondition,
+ intervalJoin.getJoinType,
+ intervalJoin.getHints)
+ createNewNode(regularJoin, children, providedTrait, requiredTrait,
requester)
Review Comment:
It might be that this approach has one major issue. What is the data type of
the timestamp column? Is it still marked as a time attribute or is it a regular
column? If it is still marked as a time attribute, windows or subsequent
interval joins would still be allowed by the planner. Of course the output
would be garbage in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
