vinishjail97 opened a new pull request, #20244: URL: https://github.com/apache/hudi/pull/20244
### Describe the issue this Pull Request addresses With the Java engine, building a metadata table index for an existing table loses most of its keys when the index has more than one file group. `JavaHoodieBackedTableMetadataWriter.bulkCommit` ignores the file group index parser and uses `JavaHoodieMetadataBulkInsertPartitioner`, which takes the file id prefix of the first record and writes every record into that one file group. Lookups hash a key to its own file group, so they find only the keys that hash to the file group that was written. This affects any index the Java engine bulk builds from existing data, for example the record index enabled on an existing table, or a secondary index built with the indexer. Found while building secondary indexes for externally written parquet files in Apache XTable: with 10 file groups, a lookup of 150 keys returned 15. ### Summary and Changelog The Java metadata bulk insert now writes each file group's records into that file group, as the Spark engine does. - `JavaHoodieMetadataBulkInsertPartitioner` sorts records by file group and then by record key (UTF-8 order as before), and gives each file group a new `CreateHandleFactory`, so the first file of every file group keeps the `-0` index the metadata table initialized it with. - `JavaBulkInsertHelper` writes each file group's run of records with that file group's file id prefix. - Tests: `TestJavaHoodieMetadataBulkInsertPartitioner#repartitionRecordsGroupsRecordsByFileGroup`, and `TestJavaHoodieBackedMetadata#testRecordIndexBuiltForExistingTableUsesEveryFileGroup`, which enables a 3 file group record index on an existing table and looks up every key. Without the fix the lookup misses about two thirds of the keys. ### Impact Java engine metadata indexes built from existing data are complete. No API or config change. ### Risk Level low. The change is limited to the Java engine's metadata table bulk insert, and a partition with one file group, such as `files`, is written exactly as before. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
