cshuo opened a new issue, #19440:
URL: https://github.com/apache/hudi/issues/19440
### Task Description
**What needs to be done:**
Support safe built-in Spark clustering and bucket rescale operations for
LSM-layout data tables.
**Why this task is needed:**
Clustering rewrites base files and may use custom-column, spatial-curve,
binary-copy, or bucket-specific execution strategies. Any strategy that emits
records in a non-record-key order breaks the LSM sorted-run invariant.
### Implementation Scope
- Support the built-in Spark sort-and-size and single-file clustering
strategies for COW and MOR LSM tables.
- Reuse table-aware LSM RDD/Row partitioners when clustering has no
conflicting user sort.
- Support consistent-bucket clustering and consistent-hashing split/merge
(`BUCKET_RESCALE`) while preserving bucket routing and file-id semantics.
- For consistent-hashing operations:
- Split an ordered input and retain ordered subsequences per output bucket.
- K-way merge multiple sorted input file groups before routing a bucket
merge; do not concatenate overlapping sorted runs.
- Disable binary/stream-copy optimization for LSM tables and fall back to an
ordered rewrite.
- Fail before clustering becomes inflight for custom sort columns, Z-order,
Hilbert, or unknown custom execution strategies whose ordering cannot be
verified.
- Keep clustering replace-commit and input file-group replacement semantics
unchanged.
### Acceptance Criteria
- [ ] Built-in RDD and Row clustering produces format-aware sorted output
files.
- [ ] COW and MOR snapshots remain correct after clustering and after a
subsequent upsert.
- [ ] Binary/stream-copy strategies fall back to a sorted rewrite for LSM
tables.
- [ ] Consistent-hashing split and merge preserve routing, file IDs, and
sorted output.
- [ ] Conflicting or unverifiable strategies fail before inflight.
- [ ] Replace-commit metadata contains the expected replaced file groups.
- [ ] Targeted clustering tests, Spark DataSource E2E, compile,
checkstyle/scalastyle, and `git diff --check` pass.
### Related Issues
**Parent feature issue:** #19065
**Depends on:** #19436, #19438, #19439
**Out of scope:** bootstrap and metadata-table index operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
