[
https://issues.apache.org/jira/browse/FLINK-41533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo de Morais updated FLINK-41533:
--------------------------------------
Fix Version/s: kafka-5.1.0
Release Note: Deletes from the upsert-kafka source (Kafka tombstones) only
contain the key, but the source used to report them as full rows. On Flink 2.1
and later, queries that need the deleted row, such as TO_CHANGELOG, could
receive deletes with NULL values. The source now reports key-only deletes, and
Flink restores the full row where it is needed.
> Upsert-kafka source declares full deletes but emits key-only deletes
> --------------------------------------------------------------------
>
> Key: FLINK-41533
> URL: https://issues.apache.org/jira/browse/FLINK-41533
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Gustavo de Morais
> Assignee: Gustavo de Morais
> Priority: Major
> Labels: pull-request-available
> Fix For: kafka-5.1.0
>
>
> The upsert-kafka source declares its changelog mode as {{UPDATE_AFTER,
> DELETE}} without {{keyOnlyDeletes(true)}}, so the planner treats its deletes
> as full deletes. At runtime a Kafka tombstone becomes a DELETE row with only
> the key fields set and all value columns NULL.
> Since FLINK-37475 the planner drops the {{ChangelogNormalize}} when the
> source already provides the delete kind a consumer requires. For upsert-kafka
> this means consumers that require full deletes receive key-only deletes with
> NULL values. For example, a row-semantic {{TO_CHANGELOG}} with an
> {{op_mapping}} that maps DELETE but not UPDATE_BEFORE emits DELETE rows with
> NULL values instead of the last row.
> We should declare {{keyOnlyDeletes(true)}} on the upsert-kafka source, so the
> planner keeps the {{ChangelogNormalize}} where full deletes are required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)