cshuo opened a new issue, #19438:
URL: https://github.com/apache/hudi/issues/19438
### Task Description
**What needs to be done:**
Support non-bucket Spark bulk insert, bulk-insert-prepped, and Dataset Row
writer overwrite paths for LSM-layout data tables.
**Why this task is needed:**
Bulk insert and Row writer paths bypass the standard Spark commit
partitioner. They need an LSM-aware partitioner that preserves each built-in
distribution mode while guaranteeing record-key ordering inside every output
base file.
### Implementation Scope
- Add LSM bulk-insert partitioners for both `JavaRDD<HoodieRecord>` and
`Dataset<Row>`.
- Support all built-in sort modes without rewriting the configured value:
- `NONE`: keep the Spark partition count and sort each partition by
`(partition path, record key)`.
- `PARTITION_SORT`: coalesce to the target parallelism and apply the same
ordering.
- `GLOBAL_SORT`: globally sort by `(partition path, record key)`.
- `PARTITION_PATH_REPARTITION`: retain partition-path routing and sort by
record key within each output partition.
- `PARTITION_PATH_REPARTITION_AND_SORT`: retain repartition semantics and
include record key in the sort.
- Support `BULK_INSERT`, `BULK_INSERT_PREPPED`, and Row writer paths used by
`INSERT_OVERWRITE` and `INSERT_OVERWRITE_TABLE`.
- For Row writer ordering:
- HFile uses Spark UTF-8 string ordering.
- Parquet/ORC uses a temporary UTF-16BE binary sort key equivalent to Java
`String.compareTo`.
- Temporary sort expressions must not enter the output schema.
- When `hoodie.populate.meta.fields=false`, fall back from DataSource Row
writer to the HoodieRecord path; reject direct Row executor calls with a clear
error.
- Fail before requested-to-inflight transition for user-defined,
custom-column, Z-order, Hilbert, or otherwise unverifiable partitioners.
### Acceptance Criteria
- [ ] RDD and Row paths produce sorted LSM base files for all five built-in
sort modes.
- [ ] `NONE` preserves input Spark partition count.
- [ ] Partition-path repartition modes retain their routing semantics.
- [ ] Both overwrite operations retain replace-commit behavior.
- [ ] UTF-8/UTF-16 distinguishing keys are physically ordered as expected.
- [ ] Unsupported custom sorting fails before the instant becomes inflight.
- [ ] Targeted partitioner/DataSource tests, compile, checkstyle/scalastyle,
and `git diff --check` pass.
### Related Issues
**Parent feature issue:** #19065
**Depends on:** #19436, #19437
**Out of scope:** bucket index and clustering paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
