CryoThrust commented on issue #13223: URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6935463008
The first Hudi replica results point to a table-commit prerequisite, beyond routing keys. [Harness and run instructions](https://gist.github.com/CryoThrust/1e9936800772d59f5358627a316f2a15). This exercises the current `HudiRecordWriter` and actual direct/cached client providers with **manually assigned stable key lanes**, not the engine's routing implementation. Hudi sources match `dev` at `37309e1a8`. The tested configuration is COPY_ON_WRITE / UPSERT, default BLOOM index; the clients report `SINGLE_WRITER`. Two tables each receive 24 composite keys across two partitions, followed by updates, deletes and reinserts. Every lane is used, and each key stays on the same lane. After every phase, the completed-timeline Parquet snapshot is compared with the complete expected key/value map. | Client path | Serial flush, replicas 1 / 2 / 4 | Concurrent flush, replicas 2 / 4 | | --- | --- | --- | | Direct | All pass | Both fail | | Resource-manager proxy | All pass | Both fail | The initial concurrent runs fail with missing `.commit.requested` files during requested-to-inflight transition, or rollback errors reporting later inflight commits. More concerningly, an additional replica-2 run for **each** client path returns from all four flush phases without throwing and has eight completed commits per table, yet the final snapshot contains **18 keys instead of 20**: two reinserted keys are missing. A fresh-process read after closing all clients confirms the mismatch in the first table for both fixtures. The serial replica-4 cached control still matches both tables after reopening. These observations do not identify the exact race or establish that every underlying Hudi WriteStatus was successful. I will keep this configuration `UNSUPPORTED`; adding `primaryKey()` alone is insufficient evidence for `KEYED`. My proposed next step is a separate, narrowly scoped investigation of table-wide commit coordination versus Hudi's supported multiwriter concurrency/locking requirements, including the nonthrowing snapshot mismatch. I would not fold that design change into a capability declaration without agreement and destination/recovery validation. INSERT/BULK_INSERT, MOR, distributed storage and checkpoint recovery remain unverified. This audit does not take over the exactly-once work or claim that a repair is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
