ravjot28 opened a new pull request, #20614:
URL: https://github.com/apache/hudi/pull/20614
### Describe the issue this Pull Request addresses
Closes #20608
On the Java engine, insert overwrite uses `JavaUpsertPartitioner`, which
packs the new records into small existing
base files. The replace commit then marks every existing file group of the
written partitions as replaced, including
the ones that received the new records, so those records are hidden once the
commit completes. The Java write client
has no insert overwrite method; the path is reached through
`HoodieJavaTable#insertOverwrite` and
`#insertOverwriteTable`.
### Summary and Changelog
Insert overwrite on the Java engine writes its records only into new file
groups, as on Spark.
- Add `JavaInsertOverwritePartitioner`, the Java counterpart of
`SparkInsertOverwritePartitioner`: a
`JavaUpsertPartitioner` that reports no small files.
- `JavaInsertOverwriteCommitActionExecutor#getInsertPartitioner` returns it.
Insert overwrite is not a
record-changing operation, so `BaseJavaCommitActionExecutor` asks for the
insert partitioner;
`JavaInsertOverwriteTableCommitActionExecutor` extends the executor and
gets the same partitioner.
- New test `TestJavaInsertOverwriteCommitActionExecutor`: `insertOverwrite`
and `insertOverwriteTable`, on
copy-on-write and merge-on-read tables.
### Impact
No API or config change. Insert overwrite no longer fills small file groups
of the partitions it replaces, so its
records stay visible after the commit.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto `release-1.2.1` and `branch-1.2.x`, and its
test passes on `branch-1.2.x`.
### Risk Level
low. Only the insert overwrite executors change, and the change follows the
Spark partitioner.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources, with
`mvn test -pl hudi-client/hudi-java-client -am -Dtest=...` and the classes
listed below:
- Without the main code change: all four cases of
`TestJavaInsertOverwriteCommitActionExecutor#testInsertOverwriteDoesNotWriteIntoReplacedSmallFile`
(`insertOverwrite` and `insertOverwriteTable`, copy-on-write and
merge-on-read) fail with "Insert overwrite wrote
into file group <id>, which it replaces".
- With the change: `TestJavaInsertOverwriteCommitActionExecutor` 4/4,
`TestJavaCopyOnWriteActionExecutor` 8/8 and
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]