ravjot28 opened a new pull request, #20589:
URL: https://github.com/apache/hudi/pull/20589
### Describe the issue this Pull Request addresses
Closes #20579
`TimelineDiffHelper` lists as new every instant of the new timeline that is
not equal to one of the old timeline, and
instant equality includes the state. A pending compaction that moves from
requested to inflight, and a pending log
compaction on every sync (the old side of the comparison holds no log
compactions), are added again by the incremental
sync; the duplicate file group check throws and the view is rebuilt from
scratch, dropping every loaded partition.
### Summary and Changelog
A file system view with incremental timeline sync stays incremental while
compactions and log compactions are pending.
- `TimelineDiffHelper#getNewInstantsForIncrementalSync`: leaves out of the
new instants every pending compaction or log
compaction that was already pending in the old timeline and is still
pending (its transition is computed from the
existing pending transitions). Finished, removed and new pending instants
keep their handling; the duplicate file
group check stays strict for a different instant.
- New test class `TestIncrementalFSViewSyncTransitions` (in-memory,
spillable and RocksDB views):
`testCompactionStateTransitionsSyncIncrementally`,
`testLogCompactionStateTransitionSyncsIncrementally`,
`testPendingLogCompactionStaysOutOfLaterSyncs`,
`testCompactionAndLogCompactionPendingTogetherSyncIncrementally`,
`testViewCreatedWhileALogCompactionIsPendingRebuildsOnce`,
`testUnschedulingAPendingCompactionRebuildsTheView`,
`testDifferentCompactionInstantForAPendingFileGroupIsRefused`. The tests
check initialization counts, pending
operations, completion, unscheduling and duplicate rejection; the
compaction transition, the unrelated delta commit
and the combined compaction and log compaction cases also compare against
a fresh view.
`testViewCreatedWhileALogCompactionIsPendingRebuildsOnce` covers a view
created while a log compaction is already
pending: its first sync still rebuilds once, and later syncs are
incremental (without this change every later sync
rebuilds as well). The last two pin the behaviour that does not change: an
unscheduled compaction still rebuilds, a
different instant for a pending file group is still refused.
- The combined case runs a compaction and a log compaction pending at the
same time on two file groups of one
partition: both move to inflight, a delta commit lands on the log
compacted file group, the compaction completes,
and the view stays incremental and equal to a fresh one. The older
`TestIncrementalFSViewSync#testAsyncMajorAndMinorCompaction` is disabled
on master for an unrelated reason and is not
enabled here.
### Impact
No API or configuration change. Only views with
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default
`false`) change: they keep their loaded partitions when a compaction moves
to inflight or while a log compaction is
pending.
A view created while a log compaction is already pending still rebuilds
once, on its first sync: the view manager
builds views with `filterCompletedAndCompactionInstants()`, which holds no
log compactions, so the old side of the
first diff does not know the instant. Later syncs are incremental.
The rebuilds removed here currently repair two other defects of
incrementally synced views, fixed in separate pull
requests: the RocksDB view can place a log file in another file slice than a
listing does
(#20587), and file groups of untouched partitions keep the timeline of their
last rebuild
(#20588). Please merge #20587 before this one or
together with it: the rebuild removed here currently repairs its incorrect
log file placement. Merging
#20588 first is recommended as well, so that this change does not extend the
retention of earlier timelines; that one affects memory only.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto
`release-1.2.1`.
### Risk Level
medium until the prerequisite view fixes are merged. This change removes
rebuilds that currently repair incorrect
RocksDB log file placement and release retained earlier timelines. After
those fixes land, it only suppresses
re-adding pending operations already held by the view.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (scalastyle did not execute
in this Java-only reactor), with `mvn test -pl hudi-hadoop-common -am
-Dtest=...` and the classes listed below:
- Without the main code change: 15 of the 21 cases of
`TestIncrementalFSViewSyncTransitions` fail on initialization
counts (five methods, three view types):
`testCompactionStateTransitionsSyncIncrementally`,
`testLogCompactionStateTransitionSyncsIncrementally`,
`testPendingLogCompactionStaysOutOfLaterSyncs`,
`testCompactionAndLogCompactionPendingTogetherSyncIncrementally` (expected
1, was 2) and
`testViewCreatedWhileALogCompactionIsPendingRebuildsOnce` (expected 2, was
3).
- With the change: `TestIncrementalFSViewSyncTransitions` 21/21,
`TestIncrementalFSViewSync` 5/5,
`TestRocksDBBasedIncrementalFSViewSync` 5/5,
`TestSpillableMapBasedIncrementalFSViewSync` 5/5 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]