[ https://issues.apache.org/jira/browse/KAFKA-5731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125090#comment-16125090 ]
ASF GitHub Bot commented on KAFKA-5731: --------------------------------------- GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/3662 KAFKA-5731 Corrected how the sink task worker updates the last committed offsets Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with. This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rhauch/kafka kafka-5731 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3662.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3662 ---- commit be34cf05fe6956a24b77b35a689bda7b55e7de12 Author: Randall Hauch <rha...@gmail.com> Date: 2017-08-12T00:42:06Z KAFKA-5731 Corrected how the sink task worker updates the last committed offsets Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with. This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks. ---- > Connect WorkerSinkTask out of order offset commit can lead to inconsistent > state > -------------------------------------------------------------------------------- > > Key: KAFKA-5731 > URL: https://issues.apache.org/jira/browse/KAFKA-5731 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Reporter: Jason Gustafson > Assignee: Randall Hauch > Fix For: 1.0.0 > > > In Connect's WorkerSinkTask, we do sequence number validation to ensure that > offset commits are handled in the right order > (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L199). > > Unfortunately, for asynchronous commits, the {{lastCommittedOffsets}} field > is overridden regardless of this sequence check as long as the response had > no error > (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L284). > > Hence if we get an out of order commit, then the internal state will be > inconsistent. To fix this, we should only override {{lastCommittedOffsets}} > after sequence validation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)