[
https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727444#comment-17727444
]
Sagar Rao commented on KAFKA-15018:
-----------------------------------
Thanks [~yash.mayya] . I have assigned this to myself. I went through your 2
approaches and feel that option 1 (the option also mentioned by Chris) might be
the cleaner one to handle this case. While the second option is viable, it
would need some more changes to have it incorporated correctly- as you pointed
out.
> Potential tombstone offsets corruption for exactly-once source connectors
> -------------------------------------------------------------------------
>
> Key: KAFKA-15018
> URL: https://issues.apache.org/jira/browse/KAFKA-15018
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 3.3.0, 3.4.0, 3.3.1, 3.3.2, 3.5.0, 3.4.1
> Reporter: Chris Egerton
> Assignee: Sagar Rao
> Priority: Major
>
> When exactly-once support is enabled for source connectors, source offsets
> can potentially be written to two different offsets topics: a topic specific
> to the connector, and the global offsets topic (which was used for all
> connectors prior to KIP-618 / version 3.3.0).
> Precedence is given to offsets in the per-connector offsets topic, but if
> none are found for a given partition, then the global offsets topic is used
> as a fallback.
> When committing offsets, a transaction is used to ensure that source records
> and source offsets are written to the Kafka cluster targeted by the source
> connector. This transaction only includes the connector-specific offsets
> topic. Writes to the global offsets topic take place after writes to the
> connector-specific offsets topic have completed successfully, and if they
> fail, a warning message is logged, but no other action is taken.
> Normally, this ensures that, for offsets committed by exactly-once-supported
> source connectors, the per-connector offsets topic is at least as up-to-date
> as the global offsets topic, and sometimes even ahead.
> However, for tombstone offsets, we lose that guarantee. If a tombstone offset
> is successfully written to the per-connector offsets topic, but cannot be
> written to the global offsets topic, then the global offsets topic will still
> contain that source offset, but the per-connector topic will not. Due to the
> fallback-on-global logic used by the worker, if a task requests offsets for
> one of the tombstoned partitions, the worker will provide it with the offsets
> present in the global offsets topic, instead of indicating to the task that
> no offsets can be found.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
