danny0405 opened a new pull request, #19079: URL: https://github.com/apache/hudi/pull/19079
### Describe the issue this Pull Request addresses This closes #19078 LSM storage layout write paths require records to be ordered by record key so native log appends and LSM file-group reader merges can preserve sorted output. The existing Java and Flink write paths did not consistently enforce that ordering before records reached native log append or file-group-reader merge handles. This PR wires LSM-aware sorted input handling into the Java and Flink write paths, and adds LSM-specific file-group-reader merge handles for COW updates, MOR small-file bin-packing merges, and compaction. It also routes CDC generated from LSM file-group-reader merges to native CDC log files instead of inline CDC log blocks. ### Summary and Changelog LSM layout tables now use sorted record-key inputs and LSM-aware merge handles across Java and Flink write paths, with native CDC log writing for LSM file-group-reader merges. - Added `LsmFileGroupReaderBasedMergeHandle` and Flink variants `FlinkLsmFileGroupReaderBasedMergeHandle` and `FlinkLsmFileGroupReaderBasedIncrementalMergeHandle`. - Updated merge-handle selection so LSM layout uses the LSM file-group-reader merge handle for writes and compaction. - Updated Flink write buffering to create record-key comparator buffers through `BufferUtils.createBuffer(...)` and sort each `RowDataBucket` with `QuickSort` before converting rows to `HoodieRecord`s. - Added Java write-path record-key sorting before native log appends when `table.requireSortedRecords()` is true. - Updated `HoodieTable.requireSortedRecords()` so LSM layout requires sorted records. - Added `HoodieNativeCDCLogger` and generalized `HoodieCDCLogWriter` so LSM file-group-reader merges can write native CDC log files from `BufferedRecord` engine rows. - Updated LSM reader/log iterator utilities to support sorted native log and CDC handling. - Added coverage in `TestHoodieMergeHandleFactory` and `TestHoodieRecordUtils`. ### Impact This affects LSM storage layout write paths in Java and Flink, including native log appends, file-group-reader-based COW merges, MOR small-file bin-packing merges, and compaction. Non-LSM layouts continue to use the existing inline log and merge-handle paths. There is no new public config. Existing LSM layout users should see sorted native log/merge behavior and native CDC log files for LSM CDC generated by file-group-reader merges. ### Risk Level medium The change touches core write-path selection, Flink buffering, Java native log append ordering, compaction merge behavior, and CDC logging. Risk is mitigated by targeted tests for merge-handle selection and record-key sorting, plus successful local validation: `mvn -pl hudi-client/hudi-client-common -DskipTests -DskipITs -Drat.skip=true -Dcheckstyle.skip=true compile` Full Flink and end-to-end LSM write/compaction suites were not run locally. ### Documentation Update none No new user-facing configuration is introduced. The behavior is gated by the existing LSM table storage layout configuration. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
