lincoln lee created FLINK-28566:
-----------------------------------
Summary: Adds materialization support to eliminate the non
determinism generated by lookup join node
Key: FLINK-28566
URL: https://issues.apache.org/jira/browse/FLINK-28566
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: lincoln lee
Fix For: 1.16.0
In order to minimize the potential exceptions or data errors when many users
use the update stream to lookup join an external
table (essentially due to the non-deterministic result based on processing-time
to lookup external tables).
When update exists in the input stream and the lookup key does not contain the
primary key of the external table,
FLINK automatically adds materialization of the update by default, so that it
will only lookup the external table
when the insert or update_after message arrives, and when the delete or
update_before message arrives, it will
directly querying the latest version of the locally materialized data and sent
it to downstream operator.
To do so，we introduce a new option 'table.exec.lookup-join.upsert-materialize'
and resue the `UpsertMaterialize`. By default, the materialize operator will be
added when an update stream lookup an external table without containing its
primary keys(includes no primary key defined). You can also choose no
materialization(NONE) or force materialization(FORCE) which will always enable
materialization except input is insert only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
