cshuo opened a new pull request, #19727: URL: https://github.com/apache/hudi/pull/19727
### Describe the issue this Pull Request addresses Closes #19439. Spark bucket-index partitioners own file-group routing and file-id assignment, so LSM tables cannot replace them with the generic LSM bulk-insert partitioners. The existing bucket paths therefore need to preserve their routing behavior while guaranteeing UTF-8 record-key ordering inside every output bucket. ### Summary and Changelog - Sort records within each LSM bucket by the full record key using UTF-8 ordering while preserving existing simple and consistent-hashing bucket routing and file-id assignment. - Enable the ordering for both RDD and Dataset Row bulk-insert paths, including the Dataset bucket-rescale path, without changing non-LSM ordering behavior or the Row schema. - Reject custom bucket sort columns for LSM tables before writes reach the inflight state. - Add unit and functional coverage for non-ASCII record keys, COW and MOR tables, simple and consistent-hashing bucket indexes, RDD and Row writers, subsequent upserts, and file-id stability. ### Impact This enables Spark LSM tables to use simple and consistent-hashing bucket indexes. It introduces no new public API or configuration. LSM Dataset Row simple-bucket writes now include the record key in the shuffle sort key, adding the key copy and comparisons required to maintain the LSM physical ordering invariant; other table layouts retain their existing behavior. ### Risk Level Medium. The change affects Spark bucket-index write partitioning and ordering. The risk is mitigated by 14 passing partitioner tests and 7 passing targeted functional cases covering routing, UTF-8 ordering, schema preservation, custom-sort rejection, file-id stability, and snapshot correctness after upsert. Compilation, the relevant Checkstyle and Scalastyle checks, and `git diff --check` also completed without violations. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
