codope opened a new pull request, #5837:
URL: https://github.com/apache/hudi/pull/5837
## What is the purpose of the pull request
This PR builds on top of #5350
- So far, archival will stop at the first savepoint commit and will not
proceed further. For users who may not be interested in incremental queries,
but just "as of instant", we can let them proceed further by skipping just the
savepoint commit. This opens up new opportunities for Hudi users. For eg, one
can retain commits for years, by adding one savepoint per day for older commits
(say > 30 days old). And they can query hudi using "as of instant" for very
old data. If not, one has to retain every commit and let archival stop at the
first commit which may not be a good experience for users.
- Had to fix one the core methods in HoodieTimeline with this change boolean
`isBeforeTimelineStarts(String instant)`. Please do check the changes here.
Prior to this patch, we don't allow any holes in the timeline. An instant is
considered committed if it's part of the active timeline or if it's < first
entry in the active timeline. But since we are letting archival go beyond
savepoints, there could be holes in the active timeline.
For eg; C1, C2, C3, Savepoint_C3, C4, C5, Savepoint_C5, C6, C7, C8, C9.
Let's say, C1, C2, C4, and C6 are archived (with the fix in this patch,
otherwise, archival will not proceed after C2).
So, the active timeline is C3, savepoint_C3, C5, Savepoint_C5, C7, C8, C9.
If a filegroup committed with C4 is checked for
`isBeforeTimelineStarts(String instant)`, we might return false. So, the fix is
to find the first non-savepoint commit in the active timeline and treat that as
the first entry in the active timeline. Any instant < this first non-savepoint
commit will be considered a valid instant time.
So, in the above case, it's C7, `isBeforeTimelineStarts(C4)` will return
true.
## Brief change log
- Added a new config named `hoodie.archive.beyond.savepoint` which will
guard this behaviour. Have set the default value to false to retain the old
behaviour.
- Changes in `HoodieTimelineArchiver` to honour the config.
- Fixed implementation of `isBeforeTimelineStarts(String instant)` to cater
to holes in the active timeline to deduce valid completed commits.
## Verify this pull request
This change added tests and can be verified as follows:
- TestHoodieTimelineArchiver#testSavepointsWithArchival
- TestHoodieActiveTimeline#testTimelineWithSavepointAndHoles
- TestHoodieFileGroup#testCommittedFileSlicesWithSavepointAndHoles
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
