vinishjail97 opened a new pull request, #19869: URL: https://github.com/apache/hudi/pull/19869
### Change Logs Tables that register parquet files written by other systems (for example tables converted with Apache XTable) record those files with the `_hudiext` marker and commit them through replace commits. The record index (RLI) and the secondary index (SI) did not work for such files: - The files carry no `_hoodie_record_key` column and the table usually has no record key fields, so record key extraction failed or produced `null` keys. - The file name recorded in the commit metadata carries the external file marker, which is not part of the name on storage, so reading the base file back for RLI failed. - The replaced file groups of a replace commit with an unknown operation type were never removed from either index. - On the first commit of a fresh table there is no completed commit to resolve the table schema from, so SI generation failed. This PR makes both indexes work for external parquet files: - **Generated record keys.** A row without a record key is keyed by the file path relative to the table base path and the row position, e.g. `americas/brazil/file_1.parquet_12`. `ExternalFilePathUtil.generateRecordKeyForRow` is the single place that builds the key so RLI and SI agree, which lets a secondary key lookup resolve to the file and row position. - **`ParquetUtils.filterRowKeys(storage, filePath, basePath, filter)`** and `FileFormatUtils.readRowKeys(storage, filePath, basePath)` generate the key for rows without a record key. The existing three argument methods keep their behavior; a missing record key without a base path now fails with a clear message instead of an NPE. - **`BaseFileRecordParsingUtils`** resolves external file names to their path on storage through `ExternalFilePathUtil.getFilePathInPartition` before reading them, and gains `generateRLIMetadataHoodieRecordsForReplacedBaseFile` to produce delete records for a replaced base file. - **RLI.** `BaseRecordIndexer` handles a replace commit that is neither a table service nor an overwrite: the record keys of the replaced file groups are deleted unless the same commit wrote them again (left anti join, like `INSERT_OVERWRITE`). - **SI.** `convertWriteStatsToSecondaryIndexRecords` receives the commit metadata. It falls back to the schema of the commit when the table has no completed commit, and for a replace commit it generates delete records for the replaced file groups that were not written to again. The record key of a row is read from the meta field when the table schema has it, from the record key fields when configured, and generated otherwise. - `SecondaryIndexer` tolerates a `null` operation type. Tables with external files have file ids that are file names rather than UUIDs, so the record index needs `_hoodie.writes.fileid.encoding=1`, as the new functional test and XTable already set. ### Impact New public methods on `ExternalFilePathUtil`, `FileFormatUtils`, `ParquetUtils` and `BaseFileRecordParsingUtils`. `SecondaryIndexRecordGenerationUtils.convertWriteStatsToSecondaryIndexRecords` takes an additional `HoodieCommitMetadata` parameter. No behavior change for tables written by Hudi. ### Risk level (write none, low medium or high below) Low. The new code paths are only reached for rows without a record key and for replace commits with a non Hudi operation type. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
