GitHub user apurvam opened a pull request: https://github.com/apache/kafka/pull/2735
KAFKA-4815 : Add idempotent producer semantics This is from the KIP-98 proposal. The main points of discussion surround the correctness logic, particularly the Log class where incoming entries are validated and duplicates are dropped, and also the producer error handling to ensure that the semantics are sound from the users point of view. There is some subtlety in the idempotent producer semantics. This patch only guarantees idempotent production upto the point where an error has to be returned to the user. Once we hit a such a non-recoverable error, we can no longer guarantee message ordering nor idempotence without additional logic at the application level. In particular, if an application wants guaranteed message order without duplicates, then it needs to do the following on the error callback: # Close the producer so that no queued batches are sent. This is important for guaranteeing ordering. # Read the tail of the log to inspect the last message committed. This is important for avoiding duplicates. You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka exactly-once-idempotent-producer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2735 ---- commit 83f5b2937d368efa768be6d7df1f57d99425a252 Author: fpj <f...@apache.org> Date: 2016-10-10T16:19:52Z Add broker producer id mapping support commit 37eac83364a67eebf0e1d96552d9fe864baa3d76 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-02-14T19:12:58Z KEOS: idempotent producer pid generation (#126) Add the transaction coordinator for pid generation and management. commit 6d918d0b44302aa81627f3f2748756356614f228 Author: hachikuji <ja...@confluent.io> Date: 2017-02-23T18:59:46Z Minor fixes for test failures and consistency (#133) commit d1868602be8b3e81d98644d673a73b9f730ad586 Author: Jason Gustafson <ja...@confluent.io> Date: 2017-03-01T22:40:30Z Fix headers, new checkstyle rules, and some breakage in KafkaApis commit f2c01a71508cd5c1eaf8c03a372f7dc308af3cf2 Author: hachikuji <ja...@confluent.io> Date: 2017-03-07T00:23:11Z Avoid removal of non-expired PIDs when log cleaning (#138) commit 31ff08eaf704b9e4fe6ee492277d981b47e8016c Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-09T01:50:01Z Client side implementation of the idempotent producer. (#129) commit 439f284fd4efa56449d43f2fb5e14614689c7b80 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-09T21:23:55Z Fix build errors due to rebase commit 0a81a40258da29ed62caf745e7c1895ca7eead4c Author: hachikuji <ja...@confluent.io> Date: 2017-03-10T17:50:01Z PIDs should be expired according to the transactional id expiration setting (#139) commit 2643e5b6bb6ad6b3d1cd6b15073a92396be5693b Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-15T18:45:07Z Fix build errors due to rebase commit 848137a28b02cd085605e92726d0457b0153e1c4 Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-16T18:43:29Z Remove depependence of MemoryRecordsBuilder on TransactionState commit 15d23b65166cff6dbd7be0e925ff9ec811d7baac Author: hachikuji <ja...@confluent.io> Date: 2017-03-23T22:30:39Z A few minor improvements (#150) commit fafffa8f2445e2a559f24e5286c948c616287f9a Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2017-03-24T20:02:48Z Reset transaction state on all irrecoverable exceptions (#153) commit 637f864887e341c51f4fd7192016d2afc45bcf1e Author: Apurva Mehta <apu...@confluent.io> Date: 2017-03-24T21:08:29Z Fix build errors due to rebase ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---