ravjot28 opened a new issue, #20608:
URL: https://github.com/apache/hudi/issues/20608
### Bug Description
**What happened:**
On the Java engine, `HoodieJavaTable#insertOverwrite` (and
`insertOverwriteTable`) can pack the new records into a
small existing base file of the partition. The replace commit then marks
every existing file group of the partition
as replaced, including the one that received the new records, so after the
commit those records are not visible.
The write reports success.
**What you expected:**
After insert overwrite, the partition holds exactly the records of the
overwrite. Spark writes insert overwrite
records only into new file groups.
**Steps to reproduce:**
1. Create a copy-on-write or merge-on-read table on the Java engine with
`hoodie.parquet.small.file.limit` at its
default (100 MB).
2. `insert` 10 records into one partition; they land in one base file below
the small file limit.
3. Start a `replacecommit` and call `HoodieJavaTable#insertOverwrite` with 5
new records for the same partition, then
commit with the returned replaced file ids.
4. The write status of step 3 names the file group of step 2, which is also
in the replaced file ids. Reading the
partition returns none of the 5 records.
**Root cause (master 9a7728c5862):**
-
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:144-149`
and `:291`: the executor's partitioner is `JavaUpsertPartitioner`, for
inserts too.
-
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaUpsertPartitioner.java:153`
packs
inserts into the small files `getSmallFiles` (`:240`) returns.
-
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaInsertOverwriteCommitActionExecutor.java:66-77`
reports every latest file group of the written partitions as replaced, the
one written into included.
- Spark returns no small files for insert overwrite:
`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwritePartitioner.java:49-52`.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at 9a7728c5862), 1.2.1
**Query engine:** Java engine (`HoodieJavaTable`)
**Relevant configs:** `hoodie.parquet.small.file.limit` > 0 (default)
### Logs and Stack Trace
none: the write succeeds and the records are not visible afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]