lincoln-lil opened a new pull request, #21219:
URL: https://github.com/apache/flink/pull/21219
## What is the purpose of the change
For current implementation, the execution plan is incorrect when do event
time temporal join on an upsert source. There's two problems:
1. for an upsert source, we should not add a ChangelogNormalize node under a
temporal join input, or it will damage the versions of the version table. For
versioned tables, we use a single-temporal mechanism which relies sequencial
records of a same key to ensure the valid period of each version, so if the
ChangelogNormalize was added then an UB message will be produced based on the
previous UA or Insert message, and all the columns are totally same include
event time
2. semantically, a filter cannot be pushed into a temporal join which using
event time, otherwise, the filter may also corrupt the versioned table
## Brief change log
* add a new StreamFilterIntoJoinRule for streaming, it doesn't push a filter
into join which is a temporal join using event time.
* add a FlinkSnapshotRequirementProgram to update snapshot requirement for
its child source to avoid adding the ChangelogNormalize node in such cases
## Verifying this change
TemporalJoinTest & TemporalRowTimeJoinOperatorTest
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving): (no)
- The serializers: (no )
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
