ravjot28 opened a new pull request, #20614:
URL: https://github.com/apache/hudi/pull/20614

   ### Describe the issue this Pull Request addresses
   
   Closes #20608
   
   On the Java engine, insert overwrite uses `JavaUpsertPartitioner`, which 
packs the new records into small existing
   base files. The replace commit then marks every existing file group of the 
written partitions as replaced, including
   the ones that received the new records, so those records are hidden once the 
commit completes. The Java write client
   has no insert overwrite method; the path is reached through 
`HoodieJavaTable#insertOverwrite` and
   `#insertOverwriteTable`.
   
   ### Summary and Changelog
   
   Insert overwrite on the Java engine writes its records only into new file 
groups, as on Spark.
   
   - Add `JavaInsertOverwritePartitioner`, the Java counterpart of 
`SparkInsertOverwritePartitioner`: a
     `JavaUpsertPartitioner` that reports no small files.
   - `JavaInsertOverwriteCommitActionExecutor#getInsertPartitioner` returns it. 
Insert overwrite is not a
     record-changing operation, so `BaseJavaCommitActionExecutor` asks for the 
insert partitioner;
     `JavaInsertOverwriteTableCommitActionExecutor` extends the executor and 
gets the same partitioner.
   - New test `TestJavaInsertOverwriteCommitActionExecutor`: `insertOverwrite` 
and `insertOverwriteTable`, on
     copy-on-write and merge-on-read tables.
   
   ### Impact
   
   No API or config change. Insert overwrite no longer fills small file groups 
of the partitions it replaces, so its
   records stay visible after the commit.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto `release-1.2.1` and `branch-1.2.x`, and its 
test passes on `branch-1.2.x`.
   
   ### Risk Level
   
   low. Only the insert overwrite executors change, and the change follows the 
Spark partitioner.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources, with
   `mvn test -pl hudi-client/hudi-java-client -am -Dtest=...` and the classes 
listed below:
   - Without the main code change: all four cases of
     
`TestJavaInsertOverwriteCommitActionExecutor#testInsertOverwriteDoesNotWriteIntoReplacedSmallFile`
     (`insertOverwrite` and `insertOverwriteTable`, copy-on-write and 
merge-on-read) fail with "Insert overwrite wrote
     into file group <id>, which it replaces".
   - With the change: `TestJavaInsertOverwriteCommitActionExecutor` 4/4, 
`TestJavaCopyOnWriteActionExecutor` 8/8 and
     `TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to