pvillard31 opened a new pull request, #10908: URL: https://github.com/apache/nifi/pull/10908
# Summary NIFI-15614 - `ConsumeKafka` - Duplicate messages during consumer group rebalance When using `ConsumeKafka` with `Kafka3ConnectionService`, duplicate messages may be processed when a consumer group rebalance occurs. [NIFI-15464](https://issues.apache.org/jira/browse/NIFI-15464) addressed a related issue by deferring offset commits during rebalance. The fix stored revoked partitions in `onPartitionsRevoked()` and had the processor call `commitOffsetsForRevokedPartitions()` after its session commit. The deferred commit approach is not enough because by the time `poll()` returns and the processor attempts to commit, the consumer is no longer part of an active group. Kafka rejects the commit with `RebalanceInProgressException`, offsets are rolled back, and messages are re-consumed as duplicates. The Kafka consumer is only in a valid state to commit offsets during the `onPartitionsRevoked()` callback. Once this callback returns, the consumer's group membership is revoked and any commit attempt will fail. We need to implement synchronous offset commit inside `onPartitionsRevoked()` callback, similar to how NiFi 1.x handled rebalances in `ConsumerLease`. This requires introducing a callback mechanism to ensure the NiFi session is committed before Kafka offsets are committed, preventing both data loss and duplicates. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - [ ] Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `./mvnw clean install -P contrib-check` - [ ] JDK 21 - [ ] JDK 25 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
