ravjot28 opened a new issue, #20607:
URL: https://github.com/apache/hudi/issues/20607
### Bug Description
**What happened:**
`BaseJavaCommitActionExecutor.execute` calls `HoodieIndex#updateLocation`
twice on the same write statuses. This
executor runs insert, upsert and delete on both table types, the prepped
insert, upsert and delete on copy-on-write,
the prepped delete on merge-on-read, and insert overwrite. The built-in
indexes are not affected in their result:
`SIMPLE`, `GLOBAL_SIMPLE` and `BLOOM` return their input, and `INMEMORY`
repeats the same map updates. A user-defined
index (`hoodie.index.class`) is updated twice per write, which matters when
its update is not idempotent or calls an
external system. The index update duration recorded in the write metadata
covers only the second call.
**What you expected:**
One index update per write through this executor, as
`BaseSparkCommitActionExecutor` does.
**Steps to reproduce:**
1. Set `hoodie.index.class` to an index that counts its `updateLocation`
calls (for example a subclass of
`HoodieBloomIndex`).
2. Write with `HoodieJavaWriteClient#insert`, `upsert` or `delete`.
3. The index counts two calls per write.
**Root cause (master 9a7728c5862):**
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:125-126`
runs `updateIndex(writeStatuses, result)` and then
`updateIndexAndMaybeRunPreCommitValidations(writeStatuses, result)`.
Both call
`table.getIndex().updateLocation(HoodieListData.eager(writeStatuses), context,
table)` (`:133` and `:297`)
and store the returned statuses and the duration in the result; the second
call overwrites what the first stored.
`updateIndex` has no other caller and no override.
A separate path is not covered here: on a merge-on-read table,
`upsertPreppedRecords` runs
`JavaUpsertPreppedDeltaCommitActionExecutor`, which sends new records
through `JavaBulkInsertHelper#bulkInsert`
(index update at `JavaBulkInsertHelper.java:86`) and then updates the index
again for all statuses
(`JavaUpsertPreppedDeltaCommitActionExecutor.java:102`). It does not go
through `BaseJavaCommitActionExecutor.execute`
and needs its own change.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT); the same two calls are in
`branch-1.2.x`, `release-1.1.1`, `release-1.0.2` and `release-0.15.0`.
**Query engine:** Java write client (`hudi-java-client`).
**Relevant configs:** `hoodie.index.class` (to observe it).
### Logs and Stack Trace
None: no error is raised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]