ravjot28 opened a new pull request, #20588:
URL: https://github.com/apache/hudi/pull/20588
### Describe the issue this Pull Request addresses
Closes #20578
`HoodieFileGroup` keeps the timeline it was built with. An incremental sync
rebuilds only the partitions the synced
instants touched, so every other loaded partition kept the timeline of its
last rebuild, and a long-lived view such as
the timeline server's held one timeline object per loaded partition. The
spillable view's serializer kept the same old
timelines in a map for spilled partitions.
### Summary and Changelog
After an incremental sync every stored file group refers to the view's
current timeline, and the view holds one
timeline object.
- `HoodieFileGroup`: a copy constructor that binds the copy to another
timeline.
- `IncrementalTimelineSyncFileSystemView` / `HoodieTableFileSystemView`:
after an incremental sync the view replaces,
under its write lock, every stored file group bound to another timeline
with a copy bound to the view's timeline.
The view's timeline here is the one complete builds use
(`getVisibleCommitsAndCompactionTimeline()`); until now an
incremental sync bound each rebuilt partition to its own
`filterCompletedAndCompactionInstants()` copy. A copy, not
a setter, so that file groups already handed to readers keep the timeline
they were read with.
- `SpillableMapBasedFileSystemView` rebinds the file groups it holds in
memory (`ExternalSpillableMap#inMemoryValues`)
and never reads its disk entries for this. It builds its
`HoodieFileGroupSerializer` with the new constructor that
takes a supplier of the view's current timeline: every group read back is
bound to it and the serializer keeps no
timeline map (a `null` supplier is refused); a read back after the view
was closed fails with a message saying so.
The existing no-argument constructor keeps its behaviour.
- `AbstractTableFileSystemView` (and `getAllFileGroups()` of the in-memory
and spillable views): the reads that
traverse every stored file group (`getLatestBaseFiles()`,
`getLatestBaseFilesInRange`, `getLatestFileSliceInRange`,
`getAllFileGroups()`) are collected while the read lock is
held. A spilled file group is now bound to the view's timeline when it is
read back, so it must be read back under
the lock; a lazy stream traversed after the lock was released could also
mix the state before and after a sync. The
per-partition reads already returned copies and are unchanged.
- The RocksDB view needs no file group rebinding. It shares the collected
all-partition reads of the base class. File
groups already returned to callers keep their timeline.
- Tests: new classes `TestIncrementalFSViewSyncTimelineRebinding` and
`TestIncrementalFSViewSyncSnapshotReads`;
`TestHoodieFileGroupSerializer` gains three tests: two for the supplier
and one that refuses a null supplier.
### Impact
No API removed: `HoodieFileGroupSerializer` gains a constructor that takes a
`Supplier<HoodieTimeline>`, and
`HoodieFileGroup` a copy constructor; `ExternalSpillableMap#inMemoryValues`
is added. Memory: a view keeps one timeline
object instead of one per loaded partition. Cost: every incremental sync
copies all in-memory file groups (one
`TreeMap` and one `HoodieFileGroup` per stored group, under the write lock),
also a sync with no new instant, because
the view installs a new timeline object on every refresh; for 100,000 file
groups with three slices each this took 6
to 11 ms warm (about 100 ms cold).
The four all-partition reads above now build their result list under the
read lock for every view type, also with
incremental sync disabled. This raises peak memory for large results; the
spillable view's `getAllFileGroups()` can
materialize the groups of every spilled partition. Their callers in the code
base collect the stream anyway. The
rebinding timings above do not measure this cost.
This fix can land independently. Merging it before #20579 is recommended:
that change removes rebuilds that currently release these earlier timelines.
**Backport:** please include this fix in the next 1.2.x patch release; the
defect is present in 1.2.1. The commit
cherry-picks cleanly onto `release-1.2.1`.
### Risk Level
medium. The change touches the stored state of the in-memory and spillable
views on every incremental sync and the
read path of the all-partition reads. Mitigation: the new tests compare the
synced views with fresh ones and check a
lazy all-partition read traversed during a sync; the existing file system
view, incremental sync, spillable map and
remote view suites pass.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (scalastyle did not execute
in this Java-only reactor), with `mvn test -pl hudi-timeline-service -am
-Dtest=...` and the classes listed below:
- Without the main code change (the master serializer test, the rebinding
test view built with the no-argument
serializer): `TestIncrementalFSViewSyncTimelineRebinding` 3 of 3 fail on
the timeline identity (expected one
timeline object, was 2; a spilled group read back is not bound to the
current timeline) and
`TestIncrementalFSViewSyncSnapshotReads` 2 of 2 on a traversal that shows
two states.
- With the change: `TestHoodieFileGroupSerializer` 4/4,
`TestHoodieFileGroup` 9/9, `TestHoodieTableFileSystemView` 51/51,
`TestSpillableMapBasedFileSystemView` 51/51, `TestIncrementalFSViewSync`
5/5,
`TestSpillableMapBasedIncrementalFSViewSync` 5/5,
`TestIncrementalFSViewSyncTimelineRebinding` 3/3,
`TestIncrementalFSViewSyncSnapshotReads` 2/2, `TestExternalSpillableMap`
32/32, `TestPriorityBasedFileSystemView` 31/31,
`TestRemoteHoodieTableFileSystemView` 54/54, `TestRequestHandler` 2/2 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]