cshuo opened a new issue, #19436:
URL: https://github.com/apache/hudi/issues/19436
### Task Description
**What needs to be done:**
Enable Spark to read LSM-layout data tables end to end and make the
sorted-run ordering consistent with the physical base-file format.
**Why this task is needed:**
Spark LSM writes and reads must use the same record-key ordering. HFile
stores keys in UTF-8 byte order, while Parquet and ORC LSM runs should retain
Java `String.compareTo` (UTF-16 code-unit) ordering.
### Implementation Scope
- Select `HoodieLsmFileGroupReader` for Spark LSM data-table snapshot/MOR
reads when merge semantics and native log files permit it.
- Keep the metadata table exemption and retain the existing reader fallback
for unsupported read modes.
- Add a base-file-format-aware record-key comparator:
- HFile: unsigned UTF-8 byte order.
- Parquet/ORC: Java `String.compareTo` order.
- Apply the comparator consistently to the LSM loser tree, sorted file-group
record buffer, create handles, and sorted/LSM merge handles.
- Preserve the base-file-only fast path so physical duplicate keys are
returned without merging; once a file slice enters the merge path, records with
the same key are merged as one logical record.
- Do not introduce a new `RecordKeyOrdering` abstraction or a new
user-facing configuration.
### Acceptance Criteria
- [ ] Spark can read base-only and base-plus-native-log LSM file slices.
- [ ] Reader and writer ordering agree for each supported base-file format.
- [ ] `U+E000` and `U+20000` tests distinguish UTF-8 from UTF-16 ordering.
- [ ] Base-only duplicate keys are preserved, while the LSM merge path
merges equal keys.
- [ ] Metadata-table behavior remains unchanged.
- [ ] Targeted common/Spark reader tests, compile, checkstyle/scalastyle,
and `git diff --check` pass.
### Related Issues
**Parent feature issue:** #19065
**RFC:** RFC-103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
