ravjot28 opened a new issue, #20578:
URL: https://github.com/apache/hudi/issues/20578
### Bug Description
**What happened:**
With incremental timeline sync, an incremental sync rebuilds only the
partitions the synced instants touched and then
refreshes the view's own timeline. The file groups of every other loaded
partition keep a reference to the timeline of
their last rebuild. After many syncs that each touch a few partitions, a
long-lived view (above all the timeline
server's) holds one `HoodieTimeline` object per loaded partition, each with
its instant list and derived maps. On a
table with a long active timeline and thousands of loaded partitions these
take a large part of the heap.
`SpillableMapBasedFileSystemView` has the same problem for the partitions it
holds in memory. For spilled partitions,
`HoodieFileGroupSerializer` keeps a map from file group id to the timeline
the group was last written with and binds
the group it reads back to that timeline, so spilled partitions keep their
old timelines alive too and come back bound
to them.
A related effect: a read over every partition returned as a lazy stream and
traversed while a sync runs can mix the
state before and after the sync.
**What you expected:**
After a sync every stored file group refers to the view's current timeline.
**Steps to reproduce:**
1. Create a table with partitions `p1` and `p2`, each with completed file
groups.
2. Build a `HoodieTableFileSystemView` (or
`SpillableMapBasedFileSystemView`) with
`hoodie.filesystem.view.incr.timeline.sync.enable=true` and load both
partitions.
3. Complete a commit that writes only to `p1` and call `view.sync()`.
4. Collect the timeline objects of `view.getAllFileGroups()` by identity:
two objects; the groups of `p2` still refer
to the timeline from before the sync.
**Root cause (master d5230b69e22):**
-
`hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:71,76,93`:
a file group keeps the
timeline it was built with and its last instant.
-
`hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:92-106`:
`maySyncIncrementally` applies the new instants, which rebuilds the file
groups of the touched partitions only
(`applyDeltaFileSlicesToPartitionView`, `:406-453`), then refreshes only
the view's own timeline (`:106`).
- `IncrementalTimelineSyncFileSystemView.java:277,383,450-452`: the rebuilt
partitions are bound to
`timeline.filterCompletedAndCompactionInstants()`, a new timeline object
per partition and instant, so touched
partitions do not share the view's timeline object either.
-
`hudi-common/src/main/java/org/apache/hudi/common/serialization/HoodieFileGroupSerializer.java:42,55,65`:
the
serializer keeps a file group id to timeline map and binds read-back
groups to it; the spillable view creates it
without access to the view's timeline
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/SpillableMapBasedFileSystemView.java:100`).
Related: #12110 added the serializer's timeline map; #15497 proposes turning
incremental timeline sync on by default.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
**Query engine:** any; most visible on the timeline server (embedded or
standalone)
**Relevant configs:**
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default `false`);
in-memory or spillable
view
### Logs and Stack Trace
none (memory retention)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]