ravjot28 opened a new pull request, #20587:
URL: https://github.com/apache/hudi/pull/20587

   ### Describe the issue this Pull Request addresses
   
   Closes #20577
   
   The RocksDB file system view applies an incremental sync by adding or 
removing single files in its stored slices. After
   a compaction is scheduled, log files can end up in another slice than a 
listing assigns them to, or in two slices, and
   a rolled-back delta commit can drop the pending compaction slice. The 
in-memory and spillable views rebuild their file
   groups and are correct.
   
   ### Summary and Changelog
   
   The incrementally synced RocksDB view equals a view built by listing, in 
every order of events covered below.
   
   - `RocksDbBasedFileSystemView#applyDeltaFileSlicesToPartitionView`: for each 
changed file group, read its stored
     slices, add or remove the delta's base and log files, build the group with 
the code a listing uses, and replace the
     group's stored slices with the result. The private per-slice merge helpers 
it replaces are removed; the public
     `fetchLatestFileSliceBeforeOrOn` stays as it is.
   - New tests in `TestIncrementalFSViewSync` (run for the in-memory, spillable 
and RocksDB views through its subclasses):
     `testCompactionScheduledWhileListedDeltaCommitIsInflight`,
     `testOneSyncSeesTheCompletedDeltaCommitAndTheScheduledCompaction`,
     `testOneSyncSeesTheListedDeltaCommitCompletedAndTheScheduledCompaction`,
     `testRollbackOfADeltaCommitKeepsThePendingCompactionSlice`, 
`testCompactionScheduledAfterRollbackLeavingLogFile`.
     Each compares the synced view with a freshly built one.
   
   ### Impact
   
   No API or configuration change. Only the `EMBEDDED_KV_STORE` view with 
incremental timeline sync changes. A sync now
   reads and rewrites every stored slice of each changed file group instead of 
single slices; the cost is per changed
   file group, not per partition.
   
   #20579 removes complete rebuilds that repair this state today; please merge
   this fix first or together with it.
   
   **Backport:** please include this fix in the next 1.2.x patch release; the 
defect is present in 1.2.1. The commit does
   not cherry-pick cleanly onto `release-1.2.1`: `RocksDbBasedFileSystemView` 
conflicts with a variable rename made on
   master after 1.2.1 (`df` to `baseFile`). I can open an adapted change 
against `branch-1.2.x`.
   
   ### Risk Level
   
   low. The file group is built by the same code a listing uses; views other 
than the RocksDB view are unchanged.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (scalastyle did not execute
   in this Java-only reactor), with `mvn test -pl hudi-hadoop-common -am 
-Dtest=...` and the classes listed below:
   - Without the main code change: the five new cases fail for the RocksDB view 
only
     (`TestRocksDBBasedIncrementalFSViewSync` 5 of 10), each on a comparison 
with a freshly built view; the in-memory and
     spillable runs of the same tests pass (10/10 each).
   - With the change: `TestIncrementalFSViewSync` 10/10, 
`TestRocksDBBasedIncrementalFSViewSync` 10/10,
     `TestSpillableMapBasedIncrementalFSViewSync` 10/10, 
`TestRocksDbBasedFileSystemView` 52/52 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to