nsivabalan opened a new pull request, #20661: URL: https://github.com/apache/hudi/pull/20661
### Describe the issue this Pull Request addresses On branch-0.x, record-path clustering of a MERGE_ON_READ file slice with several log files can write deleted keys back into the clustered base file. A clustering plan lists each slice's log files in `FileSlice` order, which is newest first (`HoodieLogFile.getReverseLogFileComparator`). The 0.x record-path clustering readers (`MultipleSparkJobExecutionStrategy`, `SingleSparkJobExecutionStrategy`, `JavaExecutionStrategy`, and Flink via `ClusteringGroupInfo`) hand those paths to `HoodieMergedLogRecordScanner` unsorted, and the scanner applies log files in the order it is given. An older insert block is therefore applied after a newer delete block, and the deleted key comes back. **Not an issue on master (1.x):** clustering there reads file slices through the file group reader, and `InputSplit` re-sorts log files oldest first (`.sorted(HoodieLogFile.getLogFileComparator())`) whatever order the plan lists them in. This fix is for branch-0.x only. ### Summary and Changelog Record-path clustering on 0.x now merges log files oldest first, so keys deleted in log files stay deleted. - `ClusteringPlanStrategy.getFileSliceInfo` records delta file paths oldest first. - `ClusteringOperation.create` also sorts delta file paths oldest first, so plans already scheduled with the old order are read correctly, on every engine. - New test `TestHoodieSparkMergeOnReadTableClustering#testClusteringKeepsKeysDeletedInLogFilesDeleted` (row writer on and off): MOR table with the in-memory index, 100 inserts that land in log files, delete 10 of them, cluster, then assert the remaining key set is exactly the 90 live keys and the last completed instant is the replacecommit. Without the fix, the record path (row writer off) returns 100 keys instead of 90: all 10 deleted keys come back. The row-writer path already passed. With the fix, both cases pass. ### Impact Fixes deleted records reappearing after clustering MOR tables on 0.x. No public API or config change. ### Risk Level low. Only the order of delta file paths handed to the merged log record scanner changes, to the order compaction and snapshot reads already use. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
