cshuo opened a new issue, #19417: URL: https://github.com/apache/hudi/issues/19417
### Problem HFile-backed sorted runs use unsigned UTF-8 byte ordering, but two RFC-103 LSM merge paths currently compare record keys with Java's UTF-16 `String.compareTo` order: - `hudi-common/src/main/java/org/apache/hudi/common/table/read/lsm/LsmFileGroupRecordIterator.java:476` - `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/LsmFileGroupReaderBasedMergeHandle.java:67` The two orders differ when comparing a supplementary character with BMP characters in the `U+E000` to `U+FFFF` range. For example, UTF-8 orders `U+E000` before `U+20000`, while UTF-16 code-unit order does the reverse. ### Impact The LSM k-way merge assumes its merge comparator matches the order of every sorted input run. Mixing UTF-16 merge order with UTF-8-sorted HFile runs can prevent equal keys from becoming adjacent across runs, potentially producing incorrect merge results or dropping an update. ### Proposed fix - Use `StringUtils.UTF8_LEXICOGRAPHIC_COMPARATOR` at both comparison sites. - Add a non-ASCII LSM merge regression test containing `U+E000` and `U+20000`. - Include the same record key in multiple sorted runs and verify that the expected latest record wins. ### Context Found while reviewing #19414, following the HFile ordering fixes from #18941. This is intentionally tracked separately so the comparator allocation fix does not expand into the LSM merge path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
