CryoThrust commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6935463008

   The first Hudi replica results point to a table-commit prerequisite, beyond 
routing keys. [Harness and run 
instructions](https://gist.github.com/CryoThrust/1e9936800772d59f5358627a316f2a15).
   
   This exercises the current `HudiRecordWriter` and actual direct/cached 
client providers with **manually assigned stable key lanes**, not the engine's 
routing implementation. Hudi sources match `dev` at `37309e1a8`. The tested 
configuration is COPY_ON_WRITE / UPSERT, default BLOOM index; the clients 
report `SINGLE_WRITER`.
   
   Two tables each receive 24 composite keys across two partitions, followed by 
updates, deletes and reinserts. Every lane is used, and each key stays on the 
same lane. After every phase, the completed-timeline Parquet snapshot is 
compared with the complete expected key/value map.
   
   | Client path | Serial flush, replicas 1 / 2 / 4 | Concurrent flush, 
replicas 2 / 4 |
   | --- | --- | --- |
   | Direct | All pass | Both fail |
   | Resource-manager proxy | All pass | Both fail |
   
   The initial concurrent runs fail with missing `.commit.requested` files 
during requested-to-inflight transition, or rollback errors reporting later 
inflight commits. More concerningly, an additional replica-2 run for **each** 
client path returns from all four flush phases without throwing and has eight 
completed commits per table, yet the final snapshot contains **18 keys instead 
of 20**: two reinserted keys are missing. A fresh-process read after closing 
all clients confirms the mismatch in the first table for both fixtures. The 
serial replica-4 cached control still matches both tables after reopening. 
These observations do not identify the exact race or establish that every 
underlying Hudi WriteStatus was successful.
   
   I will keep this configuration `UNSUPPORTED`; adding `primaryKey()` alone is 
insufficient evidence for `KEYED`. My proposed next step is a separate, 
narrowly scoped investigation of table-wide commit coordination versus Hudi's 
supported multiwriter concurrency/locking requirements, including the 
nonthrowing snapshot mismatch. I would not fold that design change into a 
capability declaration without agreement and destination/recovery validation.
   
   INSERT/BULK_INSERT, MOR, distributed storage and checkpoint recovery remain 
unverified. This audit does not take over the exactly-once work or claim that a 
repair is ready.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to