Aleksandr Savonin created FLINK-41598:
-----------------------------------------
Summary: Overlapping transactional ID prefixes can cause data loss
in KafkaSink POOLING mode
Key: FLINK-41598
URL: https://issues.apache.org/jira/browse/FLINK-41598
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Aleksandr Savonin
With exactly-once delivery and TransactionNamingStrategy.POOLING, a KafkaSink
writer lists ongoing Kafka transactions on startup to clean up transactions
left by a previous attempt. It uses a startsWith check to identify its
transactions, so distinct transactional ID prefixes can match the same ID.
To reproduce, start a sink with the prefix "sink-2" and let it write a record
in an open transaction such as "sink-2-0-0". Before that transaction commits,
start a second sink with the prefix "sink" on the same topic. The second sink
can abort "sink-2-0-0" and fence the first producer. If the transaction was
awaiting checkpoint commit, its records can be lost from the committed Kafka
output.
The same prefix check can also prevent startup. For example, if another
producer has an open transaction named "sink-a-b", the writer tries to parse
"a" as a subtask ID and throws NumberFormatException.
Cleanup should only consider IDs that match the connector’s generated format
for a configured or restored prefix: <prefix>-<subtaskId>-<offset>. A writer
with prefix "sink" should ignore both "sink-2-0-0" and "sink-a-b" and leave
those producers able to commit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)