cshuo opened a new pull request, #19767: URL: https://github.com/apache/hudi/pull/19767
### Describe the issue this Pull Request addresses Closes #19440. Spark clustering and bucket rescale previously could not safely operate on LSM-layout tables. These operations must preserve the LSM sorted-run invariant: every output file has to remain ordered by record key using UTF-8 byte ordering. ### Summary and Changelog - Enable built-in Spark sort-and-size and single-file clustering for COW and MOR LSM tables by reading input file groups through the LSM reader and reusing the existing LSM-aware Row/RDD partitioners. - Support consistent-hashing bucket split and merge while preserving bucket routing, fixed file IDs, replacement metadata, and record-key ordering. - Extract the existing LSM loser-tree implementation into a reusable common utility and add a thin merge iterator for k-way merging sorted bucket inputs. - Make simple-bucket Row/RDD bulk insert and partition-bucket rescale sort records by UTF-8 record-key order without changing bucket routing. - Disable binary/stream-copy optimization for LSM tables and fall back to an ordered rewrite. - Reject custom sort columns, spatial layout optimization, and unverified custom execution strategies before a clustering instant transitions to inflight. - Add unit and functional coverage for UTF-8 ordering, resource cleanup, COW/MOR clustering, consistent-hashing split/merge, copy fallback, early validation, bucket routing, and replace-commit metadata. ### Impact Spark users can run supported clustering and bucket-rescale operations on LSM-layout tables while retaining the ordering invariant required by LSM readers. No configuration defaults or storage format are changed. Unsupported ordering strategies fail early, while binary-copy strategies transparently use the regular ordered rewrite path. ### Risk Level medium. The change affects Spark clustering read/write paths and bucket routing. It is covered by targeted common and Spark-client unit tests plus COW/MOR functional tests that verify sorted output, snapshot correctness, split/merge routing, file IDs, replacement metadata, and subsequent upserts. ### Documentation Update None. This enables existing clustering and bucket-rescale interfaces for LSM tables and adds no new configuration. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
