cshuo opened a new issue, #19439: URL: https://github.com/apache/hudi/issues/19439
### Task Description **What needs to be done:** Support Spark simple-bucket and consistent-hashing-bucket write paths for LSM-layout data tables. **Why this task is needed:** Bucket partitioners own file-group routing and file-id assignment, so they cannot be replaced by a generic LSM partitioner. LSM support must retain bucket behavior while adding format-aware record-key ordering inside every bucket. ### Implementation Scope - Cover RDD and Dataset Row bulk-insert paths for simple bucket index and consistent hashing bucket index. - Preserve bucket/file-group routing, output partition counts, file-id prefixes, and file-id assignment behavior. - Sort records inside each bucket by the comparator selected from the base-file format. - Reuse the existing bucket routing for standard insert/upsert/delete operations and add only the LSM-required ordering. - Reject bucket custom sort columns before the instant becomes inflight. - Keep metadata-table bucket initialization behavior unchanged. ### Acceptance Criteria - [ ] Simple and consistent-hashing bucket writes produce correctly routed, sorted output files. - [ ] RDD and Row writer paths have equivalent behavior. - [ ] Existing bucket/file-id assertions continue to pass. - [ ] HFile and Parquet/ORC ordering differences are covered. - [ ] Custom bucket sort columns fail before inflight. - [ ] Snapshot reads and a subsequent upsert remain correct. - [ ] Targeted bucket tests, compile, checkstyle/scalastyle, and `git diff --check` pass. ### Related Issues **Parent feature issue:** #19065 **Depends on:** #19438 **Out of scope:** consistent-hashing clustering split/merge, which is covered by the clustering task. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
