ravjot28 opened a new issue, #20583:
URL: https://github.com/apache/hudi/issues/20583

   ### Bug Description
   
   **What happened:**
   
   With `HoodieJavaWriteClient` on a merge-on-read table and an index that 
cannot index log files (`SIMPLE`, the Java
   client's default, `BLOOM`, ...), inserts routed to a small file group 
rewrite its base file from the previous base
   file. The updates stored in that file group's log files drop out of the 
snapshot. All writes report success, and the
   read-optimized and snapshot views agree, so nothing signals the loss.
   
   **What you expected:**
   
   The snapshot keeps the latest version of every key: inserts are not packed 
into a file group whose log files would be
   left behind, as in the Spark client.
   
   **Steps to reproduce:**
   
   1. Create a merge-on-read table with the Java client and index type `BLOOM` 
(`hoodie.parquet.small.file.limit` at its
      default, above zero).
   2. `insert` 20 records into one partition; they land in a base file below 
the small-file limit.
   3. `upsert` new versions of 5 of those records; they land in a log file of 
the same file group.
   4. `insert` 20 more records into the same partition (the same happens with 
`insertPreppedRecords`).
   5. Read the snapshot: 40 records, but the 5 updated keys carry the values of 
step 2.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/deltacommit/BaseJavaDeltaCommitActionExecutor.java:67-70`:
     `getUpsertPartitioner` takes the copy-on-write `JavaUpsertPartitioner`, 
whose `getSmallFiles`
     
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaUpsertPartitioner.java:240-262`)
     picks small files from the base-file-only view, whether or not the latest 
file slice has log files.
   - `BaseJavaDeltaCommitActionExecutor.java:73-90`: for an index that cannot 
index log files, inserts assigned to such a
     small file run the copy-on-write merge handle, which reads only the 
previous base file and writes a new base file;
     the log files stay with the previous file slice.
   - The Spark client avoids this in 
`SparkUpsertDeltaCommitPartitioner#getSmallFileCandidates`
     
(`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/SparkUpsertDeltaCommitPartitioner.java:90-117`):
     when the index cannot index log files, only file slices without log files 
and outside pending compaction are
     candidates. The Java client has no merge-on-read partitioner.
   - 
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/HoodieJavaMergeOnReadTable.java`
 does not override
     `insertPrepped`, so `insertPreppedRecords` on a merge-on-read table runs 
the copy-on-write executor with the same
     small-file selection and loss.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
   **Query engine:** Java client (`HoodieJavaWriteClient`)
   **Relevant configs:** `hoodie.table.type=MERGE_ON_READ`, 
`hoodie.index.type=SIMPLE` (default) or `BLOOM` (or another index
   that cannot index log files), `hoodie.parquet.small.file.limit` > 0 (default)
   
   ### Logs and Stack Trace
   
   none: the writes succeed; the snapshot returns the older values.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to