ad1happy2go opened a new pull request, #19305: URL: https://github.com/apache/hudi/pull/19305
### Change Logs For a `ComplexKeyGenerator` with a single record-key field, the `_hoodie_record_key` encoding changed across releases (HUDI-7001): 0.14.0 and older wrote `field:value`, while 0.14.1/0.15.0/1.0.0–1.0.2 wrote the bare `value`. A table written by one release and then written by another silently mixes both formats, so upserts stop matching existing records and create duplicates. The current guard (#17834) fails all single-field complex-keygen writes by default until the user manually picks an encoding via `hoodie.write.complex.keygen.new.encoding`. This PR makes the writer self-healing instead: - **Auto-deduction** (new config `hoodie.write.complex.keygen.auto.deduce.encoding`, default `true`): on write initialization, read one stored `_hoodie_record_key` from an existing base file, detect the table's actual encoding, and pin it on the write config so upserts keep matching. - **Caching**: the deduced value is stored in `.hoodie/.aux/complex_key_encoding`; subsequent writes read the cache instead of the parquet footer. - **New/empty tables**: default to the new encoding (bare value) instead of failing the first write. - The validation guard still applies when auto-deduction is explicitly disabled; readers are never affected. ### Impact Single-field `ComplexKeyGenerator` writes work out of the box across upgrades (no manual encoding config, no failed pipelines), while preserving the existing table's key format and preventing duplicate records. ### Risk level medium Verified by three new functional test suites (auto-deduction of both encodings, existing-table upgrade safe/danger paths, new-table defaults). Also validated end-to-end with bundle jars against tables created by genuine 0.14.0 (`field:value`) and 0.14.1 (bare value) releases: encoding detected and preserved, 100/100 records upserted in place, zero duplicates. ### Documentation Update Config docs for `hoodie.write.complex.keygen.new.encoding`, `hoodie.write.complex.keygen.auto.deduce.encoding`, and `hoodie.write.complex.keygen.validation.enable` are updated in `HoodieWriteConfig`. The deployment-guide section on the complex key generator should be updated to mention auto-deduction as the default behavior. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [ ] CI passed 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
