ravjot28 opened a new issue, #20608:
URL: https://github.com/apache/hudi/issues/20608

   ### Bug Description
   
   **What happened:**
   
   On the Java engine, `HoodieJavaTable#insertOverwrite` (and 
`insertOverwriteTable`) can pack the new records into a
   small existing base file of the partition. The replace commit then marks 
every existing file group of the partition
   as replaced, including the one that received the new records, so after the 
commit those records are not visible.
   The write reports success.
   
   **What you expected:**
   
   After insert overwrite, the partition holds exactly the records of the 
overwrite. Spark writes insert overwrite
   records only into new file groups.
   
   **Steps to reproduce:**
   
   1. Create a copy-on-write or merge-on-read table on the Java engine with 
`hoodie.parquet.small.file.limit` at its
      default (100 MB).
   2. `insert` 10 records into one partition; they land in one base file below 
the small file limit.
   3. Start a `replacecommit` and call `HoodieJavaTable#insertOverwrite` with 5 
new records for the same partition, then
      commit with the returned replaced file ids.
   4. The write status of step 3 names the file group of step 2, which is also 
in the replaced file ids. Reading the
      partition returns none of the 5 records.
   
   **Root cause (master 9a7728c5862):**
   
   - 
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/BaseJavaCommitActionExecutor.java:144-149`
     and `:291`: the executor's partitioner is `JavaUpsertPartitioner`, for 
inserts too.
   - 
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaUpsertPartitioner.java:153`
 packs
     inserts into the small files `getSmallFiles` (`:240`) returns.
   - 
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaInsertOverwriteCommitActionExecutor.java:66-77`
     reports every latest file group of the written partitions as replaced, the 
one written into included.
   - Spark returns no small files for insert overwrite:
     
`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkInsertOverwritePartitioner.java:49-52`.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at 9a7728c5862), 1.2.1
   **Query engine:** Java engine (`HoodieJavaTable`)
   **Relevant configs:** `hoodie.parquet.small.file.limit` > 0 (default)
   
   ### Logs and Stack Trace
   
   none: the write succeeds and the records are not visible afterwards.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to