Gustavo de Morais created FLINK-41533:
-----------------------------------------
Summary: Upsert-kafka source declares full deletes but emits
key-only deletes
Key: FLINK-41533
URL: https://issues.apache.org/jira/browse/FLINK-41533
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Gustavo de Morais
Assignee: Gustavo de Morais
The upsert-kafka source declares its changelog mode as {{UPDATE_AFTER, DELETE}}
without {{keyOnlyDeletes(true)}}, so the planner treats its deletes as full
deletes. At runtime a Kafka tombstone becomes a DELETE row with only the key
fields set and all value columns NULL.
Since FLINK-37475 the planner drops the {{ChangelogNormalize}} when the source
already provides the delete kind a consumer requires. For upsert-kafka this
means consumers that require full deletes receive key-only deletes with NULL
values. For example, a row-semantic {{TO_CHANGELOG}} with an {{op_mapping}}
that maps DELETE but not UPDATE_BEFORE emits DELETE rows with NULL values
instead of the last row.
We should declare {{keyOnlyDeletes(true)}} on the upsert-kafka source, so the
planner keeps the {{ChangelogNormalize}} where full deletes are required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)