Yash Mayya created KAFKA-14910:
----------------------------------
Summary: Consider cancelling ongoing alter connector offsets
requests when the connector is resumed
Key: KAFKA-14910
URL: https://issues.apache.org/jira/browse/KAFKA-14910
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Yash Mayya
See discussion here for more details -
[https://github.com/apache/kafka/pull/13465#discussion_r1164465874]
The implementation for the _*PATCH /connectors/\{connector}/offsets*_ and
_*DELETE /connectors/\{connector}/offsets*_ APIs is completely asynchronous and
the check for whether the connector is stopped will only be made at the
beginning of the request.
If the connector is resumed while the alter / reset offsets request is being
processed, this can lead to certain issues (especially with non-EoS source
connectors). For sink connectors, admin client requests to alter / reset
offsets for a consumer group will be rejected if the consumer group is active
(i.e. when the connector tasks come up). For source connectors when exactly
once support is enabled on the worker, we do a round of zombie fencing before
the tasks are brought up and this will basically disable the transactional
producer used to alter offsets (the transactional producer uses the
transactional ID for task 0 of the connector). However, for source connectors
when exactly once support is not enabled on the worker (this is the default),
there are no such safeguards. We could potentially add some interruption logic
that cancels ongoing alter / reset offset requests when a connector is resumed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
