ravjot28 opened a new issue, #20579:
URL: https://github.com/apache/hudi/issues/20579

   ### Bug Description
   
   **What happened:**
   
   With `hoodie.filesystem.view.incr.timeline.sync.enable=true`, a pending 
compaction that moves from `REQUESTED` to
   `INFLIGHT` makes the next `sync()` of the file system view throw and fall 
back to a complete rebuild:
   
   ```
   ERROR IncrementalTimelineSyncFileSystemView - Got exception trying to 
perform incremental sync. Reverting to complete sync
   java.lang.IllegalArgumentException: Duplicate FileGroupId found in pending 
compaction operations. FgId :HoodieFileGroupId{partitionPath='2018/01/01', 
fileId='...'}
       at 
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java)
       at 
org.apache.hudi.common.table.view.HoodieTableFileSystemView.addPendingCompactionOperations(HoodieTableFileSystemView.java)
       at 
org.apache.hudi.common.table.view.IncrementalTimelineSyncFileSystemView.addPendingCompactionInstant(IncrementalTimelineSyncFileSystemView.java)
   ```
   
   For log compaction it is worse: while a log compaction is pending (requested 
or inflight), every sync, including one
   triggered by an unrelated delta commit, sees it as new and rebuilds the view 
("Duplicate FileGroupId found in pending
   log compaction operations"). #15480 shows this in the 
`TestRocksDBBasedIncrementalFSViewSync` logs.
   
   A complete rebuild drops every loaded partition, so the next requests list 
them again. On a merge-on-read table with
   frequent compaction scheduling and many loaded partitions, the timeline 
server keeps rebuilding its views.
   
   **What you expected:**
   
   A pending compaction or log compaction that only changed state keeps the 
operations the view already holds, and the
   sync stays incremental; the view after the sync equals a freshly built view.
   
   **Steps to reproduce:**
   
   1. Merge-on-read table, one partition, a delta commit `001` writing log 
files for two file groups.
   2. Build a `HoodieTableFileSystemView` with incremental timeline sync and 
load the partition.
   3. Schedule a compaction `002` over the two file slices, `view.sync()`: 
incremental, two pending operations.
   4. Transition `002` from requested to inflight, `view.sync()`: the sync 
throws internally and the view is rebuilt.
   
   For log compaction: schedule a log compaction, sync, then add a delta commit 
to another partition and sync again.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineDiffHelper.java:84`:
 the new instants are
     those of the new timeline not contained in the old one, and 
`HoodieInstant.equals`
     
(`hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstant.java:86`)
 compares the state, so the
     inflight compaction is not equal to the requested one the view held.
   - `TimelineDiffHelper.java:52,64`: the old side comes from 
`filterCompletedAndCompactionInstants()`, which keeps pending
     compactions but drops pending log compactions, while line 84 iterates the 
unfiltered new timeline. A pending log
     compaction is therefore new on every sync.
   - `TimelineDiffHelper.java:69,86` already compute the transitions of the old 
pending compactions and log compactions,
     but use them only for the finished or removed lists.
   - 
`hudi-common/src/main/java/org/apache/hudi/common/table/view/HoodieTableFileSystemView.java:224-226`
 (and the log
     compaction and RocksDB counterparts) refuse a file group that is already 
pending; the exception is caught in
     `IncrementalTimelineSyncFileSystemView.java:110-115`, which clears and 
rebuilds the view.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
   **Query engine:** any engine that serves a writer through the embedded (or a 
standalone) timeline server; merge-on-read
   tables with compaction or log compaction
   **Relevant configs:** 
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default `false`)
   
   ### Logs and Stack Trace
   
   See "What happened" above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to