Cliff Rhyne created KAFKA-2725: ---------------------------------- Summary: high level consumer rebalances with auto-commit disabled should throw an exception Key: KAFKA-2725 URL: https://issues.apache.org/jira/browse/KAFKA-2725 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2.1 Environment: Experienced on Java running in linux Reporter: Cliff Rhyne
Auto-commit is a very resilient mode. Drops in zookeeper sessions due to JVM garbage collection, network, rebalance or other interference are handled gracefully within the kafka client. Systems still can drop due to unexpected gc or network behavior. My proposal is to handle this drop better when auto-commit is turned off: - If a rebalance or similar occur (which cause the offset to get reverted in the client), check and see if the client was assigned back to the same partition or a different one. If it's the same partition, find the place last consumed (it doesn't do this today for us). This is to make a graceful recovery. - If the partition assignment changes (which can mean duplicate data is getting processed), throw an exception back to the application code. This lets the application code handle this exception-case with respect to the work it's doing (with might be transactional). Failing "silently" (yes it's still getting logged) is very dangerous in our situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)