corgy-w opened a new issue, #13224: URL: https://github.com/apache/seatunnel/issues/13224
### Background #12440 adds an opt-in `semantics = EXACTLY_ONCE` two-phase commit to the Hudi sink. The review of that PR agreed that two items can be handled as follow-ups instead of blocking the merge. ### 1. Carry commit stats instead of serialized `WriteStatus` in the checkpoint state `HudiCommitInfo` currently serializes `List<WriteStatus>` into the committer checkpoint state and keeps the same objects on the writer heap until the checkpoint completes. `WriteStatus` is a Hudi client class and, when records are tracked, it can hold a `HoodieRecordDelegate` per written record, so the state size and the heap usage scale with the number of records in a checkpoint interval, and the persisted format depends on Hudi internals. The sink only needs the statistics to call `HoodieJavaWriteClient#commit`. The follow-up should carry a small connector-owned stats object (records written, errors, partition/file stats needed by the commit) and add a test with the metadata table enabled, which is the configuration that tracks records. ### 2. Add a fault-injection E2E for the exactly-once path `HudiIT#testWriteHudiWithExactlyOnceSemantics` is a short batch job with parallelism 1. It verifies that the records are invisible until the checkpoint commits, but it does not fail a task between the checkpoint completion and the commit, so it cannot prove "no loss, no duplicate" across a restart. The follow-up should add an E2E that fails the job (or a task) after the checkpoint completes and before the aggregated committer commits the instant, then restarts from the checkpoint and asserts that the records are committed exactly once. This is also the test that would catch a regression in the restore-time reclaim of orphaned instants. ### Related - #12440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
