ravjot28 opened a new issue, #20612:
URL: https://github.com/apache/hudi/issues/20612
### Bug Description
**What happened:**
On a merge-on-read table, `HoodieJavaWriteClient#upsertPreppedRecords`
leaves its instant in the requested state when
the batch holds only tagged records (updates): the following `commit` fails
with
`IllegalArgumentException: File
.../.hoodie/timeline/<instant>.deltacommit.inflight does not exist!`. When the
batch
also holds new records, the commit works, but the index's `updateLocation`
and the pre-commit validators run twice:
once for the new records' statuses, then for all statuses.
**What you expected:**
The prepped upsert moves its instant to inflight before writing, as the
other Java and Spark write paths do, and
updates the index and runs the validators once for the whole write.
**Steps to reproduce:**
1. Create a merge-on-read table with `HoodieJavaWriteClient` (index type
`BLOOM`).
2. `insert` 20 records and commit.
3. Tag new versions of those records (`HoodieIndex#tagLocation`), start a
commit and pass them to
`upsertPreppedRecords`.
4. `commit` fails: the delta commit has no inflight file.
5. With a user-defined index (`hoodie.index.class`) and a batch of tagged
updates plus new records, the index's
`updateLocation` is called twice for the write.
**Root cause (master a65a573a49f):**
- `HoodieJavaMergeOnReadTable#upsertPrepped`
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/HoodieJavaMergeOnReadTable.java:67`)
runs
`JavaUpsertPreppedDeltaCommitActionExecutor#execute`
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/deltacommit/JavaUpsertPreppedDeltaCommitActionExecutor.java:58-104`),
which writes the updates to log files (`:86`) without moving the instant
to inflight. Unlike
`BaseJavaCommitActionExecutor#execute`
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:101`),
it never saves the workload profile to the inflight instant.
- New records go to `JavaBulkInsertHelper#bulkInsert(..., executor, ...)`
(`JavaUpsertPreppedDeltaCommitActionExecutor.java:92`),
which moves the instant to inflight
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaBulkInsertHelper.java:73-78`),
after the updates were written, and calls
`updateIndexAndMaybeRunPreCommitValidations` for their statuses (`:86`);
the executor then calls it again for all statuses
(`JavaUpsertPreppedDeltaCommitActionExecutor.java:102`).
- The Java metadata table writer, the main caller of this executor, moves
its own instant to inflight before the
write
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/metadata/JavaHoodieBackedTableMetadataWriter.java:144`),
so metadata table commits are not affected; prepped upserts into a data
table are.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at a65a573a49f), 1.2.1
**Query engine:** Java client (`HoodieJavaWriteClient`)
**Relevant configs:** `hoodie.table.type=MERGE_ON_READ`; the second symptom
needs a user-defined index
(`hoodie.index.class`) or pre-commit validators to be visible
### Logs and Stack Trace
```
java.lang.IllegalArgumentException: File <base
path>/.hoodie/timeline/<instant>.deltacommit.inflight does not exist!
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.transitionStateToComplete(ActiveTimelineV2.java:566)
at
org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2.saveAsComplete(ActiveTimelineV2.java:169)
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:325)
at
org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:275)
at
org.apache.hudi.client.HoodieJavaWriteClient.commit(HoodieJavaWriteClient.java:108)
at
org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:209)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]