cshuo opened a new pull request, #19502:
URL: https://github.com/apache/hudi/pull/19502
### Describe the issue this Pull Request addresses
Closes #19436.
Spark LSM data-table reads need to use the LSM file-group reader and the
same record-key ordering as the physical base-file format. HFile keys are
ordered by unsigned UTF-8 bytes, while Parquet and ORC sorted runs retain Java
`String.compareTo` (UTF-16 code-unit) ordering.
### Summary and Changelog
- Select `HoodieLsmFileGroupReader` for eligible Spark snapshot/MOR reads of
LSM data tables.
- Preserve metadata-table behavior and the existing fallback for skip-merge
reads or unsupported log-file types.
- Add a base-file-format-aware record-key comparator:
- HFile: unsigned UTF-8 byte ordering.
- Parquet/ORC: Java `String.compareTo` ordering.
- Apply the comparator consistently to the LSM loser tree, sorted file-group
record buffer, create handles, and sorted/LSM merge handles.
- Preserve the base-file-only fast path so physical duplicate keys are
returned directly; file slices entering the merge path continue to merge equal
keys as one logical record.
- Propagate the configured table storage layout during Spark SQL table
initialization.
- Add coverage using `U+E000` and `U+20000` to distinguish UTF-8 and UTF-16
ordering.
### Impact
Spark can read eligible LSM-layout data tables end to end. Reader and writer
ordering now agree with each base-file format without imposing UTF-8 comparison
overhead on Parquet and ORC sorting. No new public API or user-facing
configuration is introduced.
### Risk Level
Medium. This changes Spark MOR reader selection for LSM-layout data tables
and comparator selection on sorted-run paths. The change retains existing
fallbacks and is covered by targeted common reader/ordering tests, Spark
datasource E2E coverage, Spark client compilation, and Spark 4 packaging
verification.
### Documentation Update
None. No configuration or public API is added or changed.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
