cshuo opened a new issue, #19437:
URL: https://github.com/apache/hudi/issues/19437
### Task Description
**What needs to be done:**
Support the standard Spark write operations and MOR compaction services for
LSM-layout data tables.
**Why this task is needed:**
Real Spark workloads mix insert, upsert, delete, and overwrite operations.
Restricting an LSM table to upsert-only behavior prevents normal ingestion
workflows such as an initial insert followed by incremental upserts.
### Implementation Scope
- Support the HoodieRecord/RDD paths for:
- `UPSERT` and `UPSERT_PREPPED`
- `INSERT` and `INSERT_PREPPED`
- `DELETE` and `DELETE_PREPPED`
- `INSERT_OVERWRITE` and `INSERT_OVERWRITE_TABLE`
- `DELETE_PARTITION`
- Sort LSM input in `BaseSparkCommitActionExecutor` using the comparator
selected from the base-file format.
- Remove the Spark LSM rejection for insert and insert-prepped while
retaining the metadata-table exemption.
- Support MOR `COMPACT` and `LOG_COMPACT`, producing sorted base files and
native log blocks.
- Keep replace-commit, file-group replacement, rollback, clean, and
index-update semantics unchanged.
- Define duplicate-key behavior explicitly:
- A base-file-only insert may preserve duplicate physical keys.
- After an upsert/delete introduces a merge path, equal keys are merged as
one logical key.
### Acceptance Criteria
- [ ] All listed operations succeed for applicable COW and MOR LSM tables.
- [ ] Every newly written base file is ordered according to its base-file
format.
- [ ] An `INSERT -> UPSERT` MOR workflow returns the latest logical records.
- [ ] A duplicate-key insert is preserved on the base-only path and
converges through a later LSM merge.
- [ ] Regular and log compaction preserve snapshot correctness and
sorted-run invariants.
- [ ] Replace commit and file-group replacement metadata remain correct.
- [ ] Targeted Spark client/DataSource tests, compile,
checkstyle/scalastyle, and `git diff --check` pass.
### Related Issues
**Parent feature issue:** #19065
**Depends on:** #19436
**Out of scope:** bulk insert, bucket-specific paths, clustering, bootstrap,
and metadata-table indexing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
