yihua opened a new pull request, #984:
URL: https://github.com/apache/hudi-rs/pull/984
## Description
An incremental read took each touched file group's slice from a snapshot at
the last admitted commit's requested time, built over every completed commit. A
commit requested before that point but completed after the window (OCC
multi-writer, or a late table service) could then supply the slice, the
commit-time mask dropped its rows, and the window's version of the row was
gone. `resolve_incremental_window` now also carries the window end as given,
and incremental planning (touched file groups, replaced file groups and file
visibility) runs on a timeline truncated to commits completed by that bound via
`Timeline::as_of_completion_time`. It is a no-op on layout v1, and snapshot and
time-travel planning are unchanged. The incremental note in AGENTS.md is
updated to match.
For regular commits this matches Hudi 1.x, whose incremental file system
view only holds files written by the admitted commits. It deliberately differs
for a replacecommit requested inside the window but completed after it: Java's
replaced-file-group set comes from the full timeline, so its answer flips once
that commit completes, while this returns the same rows either way.
Closes #922
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
A case-table test uses native writes with out-of-order completion (late
upsert on COW and MOR, late partition overwrite, late compaction) and asserts
each window returns the same rows, and `get_file_slices` plans the same slices,
before and after the late commit completes; on main the upsert and overwrite
cases lose the row. Unit tests cover `Timeline::as_of_completion_time` on both
layouts and that the window end comes only from resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]