corgy-w opened a new issue, #13224:
URL: https://github.com/apache/seatunnel/issues/13224

   ### Background
   
   #12440 adds an opt-in `semantics = EXACTLY_ONCE` two-phase commit to the 
Hudi sink. The review of that PR agreed that two items can be handled as 
follow-ups instead of blocking the merge.
   
   ### 1. Carry commit stats instead of serialized `WriteStatus` in the 
checkpoint state
   
   `HudiCommitInfo` currently serializes `List<WriteStatus>` into the committer 
checkpoint state and keeps the same objects on the writer heap until the 
checkpoint completes. `WriteStatus` is a Hudi client class and, when records 
are tracked, it can hold a `HoodieRecordDelegate` per written record, so the 
state size and the heap usage scale with the number of records in a checkpoint 
interval, and the persisted format depends on Hudi internals.
   
   The sink only needs the statistics to call `HoodieJavaWriteClient#commit`. 
The follow-up should carry a small connector-owned stats object (records 
written, errors, partition/file stats needed by the commit) and add a test with 
the metadata table enabled, which is the configuration that tracks records.
   
   ### 2. Add a fault-injection E2E for the exactly-once path
   
   `HudiIT#testWriteHudiWithExactlyOnceSemantics` is a short batch job with 
parallelism 1. It verifies that the records are invisible until the checkpoint 
commits, but it does not fail a task between the checkpoint completion and the 
commit, so it cannot prove "no loss, no duplicate" across a restart.
   
   The follow-up should add an E2E that fails the job (or a task) after the 
checkpoint completes and before the aggregated committer commits the instant, 
then restarts from the checkpoint and asserts that the records are committed 
exactly once. This is also the test that would catch a regression in the 
restore-time reclaim of orphaned instants.
   
   ### Related
   
   - #12440
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to