ravjot28 opened a new issue, #20577:
URL: https://github.com/apache/hudi/issues/20577
### Bug Description
**What happened:**
With `hoodie.filesystem.view.type=EMBEDDED_KV_STORE` and
`hoodie.filesystem.view.incr.timeline.sync.enable=true`, the
incrementally synced RocksDB view assigns log files to file slices
differently from a view built by listing:
- A log file whose delta commit has no completion time (an inflight delta
commit, or a rolled-back one whose file was
left on storage) stays in its previous file slice when a compaction is
scheduled; a listing puts it in the new
pending compaction slice. When that delta commit completes, its log file
is added to the slice its completion time
selects and also kept in the previous slice.
- When one sync sees both a completed delta commit and the compaction
scheduled while it was inflight (the usual order
when another process schedules the compaction), the log file stays in the
previous slice, although its completion
time selects the compaction slice. A compaction planned from that view
then leaves the log file out.
- When a delta commit that landed in the pending compaction slice is rolled
back, the emptied slice is dropped, while a
listing keeps it; later log files then go to the previous slice.
Any complete rebuild of the view repairs the state.
**What you expected:**
The incrementally synced RocksDB view equals a view built by listing: a log
file without a completion time belongs to
the latest file slice of its file group, and a log file belongs to exactly
one slice. The in-memory and spillable views
already behave this way.
**Steps to reproduce:**
1. Delta commits `11`, `12`, `13` write log files to every file group of a
merge-on-read table; a RocksDB view with
incremental sync syncs them.
2. Delta commit `14` writes its log files and stays inflight. Close the
first view, create a new incrementally synced
RocksDB view, and load the partition so it lists `14`'s log files before
the compaction is scheduled.
3. Schedule a compaction `C` over the latest file slices; sync the view.
4. Compare with a freshly built view: fresh `{C <- 14 | 11 <- 11,12,13}`,
synced RocksDB view
`{C <- (none) | 11 <- 11,12,13,14}`.
5. Complete `14` and sync: fresh `{C <- 14 | 11 <- 11,12,13}`, synced
RocksDB view `{C <- 14 | 11 <- 11,12,13,14}`.
**Root cause (master d5230b69e22):**
- A listing builds each file group with the pending compaction slice first
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:245-256`)
and then adds
log files; a log file without a completion time belongs to the latest slice
(`hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:167`).
- The in-memory and spillable views rebuild every file group of a partition
on each delta, so they reassign log files.
- The RocksDB view patches single slices
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/RocksDbBasedFileSystemView.java:314-377`):
a pending
compaction adds an empty slice, `shouldMergeFileSlice`
(`RocksDbBasedFileSystemView.java:585-595`) is false, and the
log files without a completion time stay in the previous slice. When the
delta commit completes, its log file is
merged into the slice its completion time selects, and nothing removes it
from the previous slice.
Related: #15480 (errors in `TestRocksDBBasedIncrementalFSViewSync` logs).
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1 (table
version 8 and above, completion-time based file
slicing)
**Query engine:** any writer served by a timeline server with the RocksDB
view
**Relevant configs:** `hoodie.filesystem.view.type=EMBEDDED_KV_STORE`,
`hoodie.filesystem.view.incr.timeline.sync.enable=true`
### Logs and Stack Trace
none: the view returns different file slices than a fresh listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]