ravjot28 opened a new pull request, #20618:
URL: https://github.com/apache/hudi/pull/20618
### Describe the issue this Pull Request addresses
Closes #20612
On a merge-on-read table, `HoodieJavaWriteClient#upsertPreppedRecords` runs
`JavaUpsertPreppedDeltaCommitActionExecutor`, which writes the updates to
log files without moving the instant from
requested to inflight. Only new records reach
`JavaBulkInsertHelper#bulkInsert(..., executor, ...)`, which makes that
transition (after the updates were written) and also updates the index and
runs the pre-commit validators for their
statuses, before the executor does both again for all statuses. A batch of
updates only cannot be committed
("deltacommit.inflight does not exist"); a batch with new records updates
the index twice. The Java metadata table
writer moves its instant to inflight itself before calling this executor, so
metadata table commits are not affected.
### Summary and Changelog
Prepped upserts on Java merge-on-read tables move their instant to inflight
before writing and update the index once.
- `JavaUpsertPreppedDeltaCommitActionExecutor#execute`:
- saves the workload profile of the prepped records to the inflight
instant before writing, as
`BaseJavaCommitActionExecutor#execute` does, unless the instant is
already inflight (the metadata table writer's
case; the same check `JavaBulkInsertHelper` makes);
- writes new records through the `JavaBulkInsertHelper#bulkInsert`
overload that returns their write statuses
(same partitioner, handle factory and parallelism as before), so
`updateIndexAndMaybeRunPreCommitValidations`
runs once, for all statuses, at the end.
- New `TestJavaUpsertPreppedDeltaCommitActionExecutor` (merge-on-read): a
prepped batch of tagged updates only
commits as a completed delta commit with the updated records readable; a
batch of tagged updates and new records
calls the index's `updateLocation` once (an index set through
`hoodie.index.class` counts the calls) and commits
all records.
### Impact
No API or config change. Prepped upserts into Java merge-on-read data tables
can be committed whatever the batch
holds, and their inflight instant carries the workload profile. User-defined
indexes and pre-commit validators see
one call per write.
**Backport:** please include this fix in the next 1.2.x patch release. On
`release-1.2.1` the commit conflicts only in
the import block of `JavaUpsertPreppedDeltaCommitActionExecutor` (1.2.1 does
not import `AppendHandleFactory`); the
code hunks and the test apply as they are.
### Risk Level
low. The inflight transition and the single index update are what the other
Java write paths already do; the
metadata table's own transition is detected and kept.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources, with
`mvn test -pl hudi-client/hudi-java-client -am -Dtest=...`:
- Without the main code change: both tests of
`TestJavaUpsertPreppedDeltaCommitActionExecutor` fail:
`testUpdatesOnlyBatchCommits` with `IllegalArgumentException: File
.../<instant>.deltacommit.inflight does not
exist!` from `commit`, `testMixedBatchUpdatesTheIndexOnce` with `expected:
<1> but was: <2>` index updates.
- With the change: `TestJavaUpsertPreppedDeltaCommitActionExecutor` 2/2,
`TestJavaHoodieBackedMetadata` 68/68 (the
metadata table writer uses this executor),
`TestHoodieJavaClientOnMergeOnReadStorage` 5/5,
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]