[
https://issues.apache.org/jira/browse/KAFKA-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399681#comment-16399681
]
ASF GitHub Bot commented on KAFKA-6661:
---------------------------------------
rhauch opened a new pull request #4716: KAFKA-6661: Ensure sink connectors
don’t resume consumer when task is paused
URL: https://github.com/apache/kafka/pull/4716
Changed WorkerSinkTaskContext to only resume the consumer topic partitions
when the connector/task is not in the paused state.
The context tracks the set of topic partitions that are explicitly
paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks
it currently resumes all topic partitions *except* those that are still
explicitly paused in the context. Therefore, the change above should result in
the desired behavior.
Several debug statements were added to record when the context is called by
the connector.
This can be backported to older releases, since this bug goes back to 0.10
or 0.9.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Sink connectors that explicitly 'resume' topic partitions can resume a paused
> task
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-6661
> URL: https://issues.apache.org/jira/browse/KAFKA-6661
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 0.9.0.0, 0.10.0.0, 0.11.0.0, 1.0.0
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Priority: Critical
>
> Sink connectors are allowed to use the {{SinkTaskContext}}'s methods to
> explicitly pause and resume topic partitions. This is useful when connectors
> need additional time processing the records for specific topic partitions
> (e.g., the external system has an outage).
> However, when the sink connector has been paused via the REST API, the worker
> for the sink tasks pause the consumer. When the connector is polled, the poll
> request might timeout and return no records. Connect then calls the task's
> {{put(...)}} method (with no records), and this allows the task to optionally
> call any of the {{SinkTaskContext}}'s pause or resume methods. If it calls
> resume, this will unexpectedly resume the paused consumer, causing the
> consumer to return messages and the connector to process those messages --
> despite the connector still being paused.
> This is reported against 1.0, but the affected code has not been changed
> since at least 0.9.0.0.
> A workaround is to remove rather than pause a connector. It's inconvenient,
> but it works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
