wombatu-kun opened a new pull request, #20316:
URL: https://github.com/apache/hudi/pull/20316
### Describe the issue this Pull Request addresses
Closes #20315
### Summary and Changelog
The Kafka Connect sink no longer rewrites committed records when an action
after the commit fails.
- `KafkaConnectTransactionServices.endCommit`: when `javaClient.commit`
throws, it reloads the timeline and, if the instant completed, logs the failure
and treats the commit as successful; otherwise it rethrows. The failures this
covers come from automatic cleaning, archiving and inline table services, which
run inside `javaClient.commit` after the instant completes. The follow-up table
service scheduling and meta sync then always run, and a failure there is logged
without failing the commit. Before, any of these exceptions made
`ConnectTransactionCoordinator` treat the completed commit as failed: it kept
its old Kafka offsets and skipped the ACK, so the next START_COMMIT sent the
participants back to those offsets and they wrote the committed records again.
- `TestKafkaConnectTransactionServices`: a meta sync failure and an inline
clustering failure after the instant completed no longer fail `endCommit`, meta
sync still runs after the clustering failure, and a commit whose instant stays
pending still fails.
### Impact
Correctness fix for the Kafka Connect sink: on COW tables, which are written
with bulk insert, each such failure duplicated a whole commit's records, and
the next commit could record lower Kafka offsets than the completed one. A
failed post-commit action is now logged at ERROR and runs again after the next
commit, so the sink effectively ignores post-commit failures regardless of
`hoodie.write.can.ignore.post.commit.failures` and
`hoodie.fail.writes.on.inline.table.service.exception`; in Kafka Connect such a
failure never stopped ingestion, it only triggered the rewrite. There are no
public API, configuration, or storage format changes.
### Risk Level
low. A failure inside `javaClient.commit` now counts as success only when
the reloaded timeline shows the instant completed; a commit that fails before
completing still throws and takes the existing recovery path. Each part of the
fix is pinned by a test that fails without it: removing the fix fails both
success tests with the original meta sync or clustering exception, dropping the
completed-instants filter or the check itself fails the pending-instant test
("nothing was thrown"), reading the cached timeline instead of reloading it
fails the clustering test, and skipping the follow-up actions after an
in-commit failure fails its meta sync count. All 25 hudi-kafka-connect unit
tests pass. An end-to-end run with the real transaction services, a meta sync
tool that always fails, and the real coordinator and participant wrote Kafka
offsets `0..94`, `95..194` and `195..294` into three completed instants with
the fix, and `0..94`, `0..89` and `0..94` without it.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
