yihua opened a new pull request, #7160:
URL: https://github.com/apache/hudi/pull/7160
### Change Logs
When multiple writers are writing to the same Hudi table under optimistic
concurrency control, one writer may start later and finish the write
transaction earlier than the other concurrent writer. In this case, the Hudi
timeline contains an inflight commit before a completed commit. As a concrete
example, writer 1 starts a commit at t1 and later writer 2 starts another
commit at t2 (t2 > t1). Commit t2 finishes earlier than t1.
```
t3 t4
---------------------------------------------------------> t
instant t1 |------------------------------| (writer 1)
instant t2 |--------------| (writer 2)
```
When Hudi incremental source is used on such a Hudi table, the logic to
determine the next fetch (`IncrSourceHelper.calculateBeginAndEndInstants()`) is
susceptible to data loss, missing the inflight instants. The method filters
for the completed instants only and uses that to determine the instants to
fetch, thus the checkpoint. For the same example, when incrementally fetching
the data at t3, the checkpoint advances to t2 and ignores t1 indefinitely.
To fix the problem, this PR adds the logic to only look at contiguous
completed commits / deltacommits / non-clustering replacecommits for
incremental pulls, e.g., completed instants before t1 in the example, so that
the checkpoint never goes over the inflight commits which can cause data loss.
To implement this, we figure out the earliest incomplete commit, deltacommit,
or non-clustering replacecommit to bound the incremental pulls.
This PR fixes the `HoodieIncrSource`, `S3EventsHoodieIncrSource`, and
`GcsEventsHoodieIncrSource` for the same issue.
### Impact
This PR avoids missing instants for incremental pulls for the scenarios
mentioned above for multi-writer.
### Risk level
medium
This PR adds a new test,
`TestHoodieIncrSource#testHoodieIncrSourceInflightCommitBeforeCompletedCommit`,
to verify the expected behavior. Before this PR, the same test failed.
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
