ad1happy2go opened a new pull request, #20203: URL: https://github.com/apache/hudi/pull/20203
### Describe the issue this Pull Request addresses closes #20202 Follows #19304. When `hoodie.table.complex.keygenerator.encoding` is missing, the encoding is deduced from the first record of the first file the newest commit wrote. A commit that packs inserts into an existing small file rewrites that file with its older records first, keyed the way their writer keyed them, so the deduction can return an older writer's encoding. ### Summary and Changelog `KeyGenUtils.deduceComplexKeyGenEncodingFromData` now reads a record the newest commit wrote: in a base file, the record with the greatest `_hoodie_commit_time` (projected with `_hoodie_record_key`, stopping at one the inspected commit wrote); in a log file, the first record of the data block that commit appended, else of the newest data block. Clustering and compaction commits (by operation type, so insert overwrite still counts) are skipped and do not count toward the bound of inspected commits, since they rewrite records other commits keyed. Within a commit, files the commit created are read first (their first record is one the commit wrote), then smaller files, so a full scan of a rewritten file is the fallback rather than the norm. The same change is on the branch-0.x port, #20200. ### Impact None for tables whose data carries a single encoding. For a table whose history spans both key formats, the recorded encoding follows the newest writer instead of whichever record a rewritten file happens to start with. ### Risk Level low. Only the encoding deduction changes; it runs only while the property is missing (writers then record it, and readers use it for record key pruning). New tests reproduce the small file and clustering cases: both fail without the change (`FIELD_PREFIXED`) and pass with it (`VALUE_ONLY`). ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
