fhan688 opened a new pull request, #19949:
URL: https://github.com/apache/hudi/pull/19949
### Describe the issue this Pull Request addresses
Part of #19286.
For pre-8 MOR tables, Flink incremental queries may fall back to scanning
the latest file slice after resolving the query range from completion time to
requested time.
In this case, the selected file slice can be newer than the query end
instant. The existing split generation logic uses the earlier of the file-slice
instant and query end as the log reader's high watermark.
This can prevent the reader from loading the complete file slice and cause
records within the requested incremental range to be missed.
The log reader's physical high watermark and the query's logical instant
range serve different purposes: the reader must cover the selected file slice,
while `InstantRange` continues to exclude records beyond the query end.
### Summary and Changelog
- Use the later of the file-slice instant and query end as the MOR log
reader high watermark.
- Keep `InstantRange` as the logical boundary for filtering records
outside the incremental query range.
- Update the existing input-split test to verify that the high watermark
covers both boundaries.
- Add a regression test for an archived incremental range on a
table-version-6 MOR table.
- Verify that log blocks after the query end can be physically read but do
not participate in the final merged result.
### Impact
Fixes Flink incremental-read correctness for pre-8 MOR tables when the
selected file slice advances beyond the query end instant.
There are no public API, configuration, or storage-format changes.
Affected queries may physically scan log blocks after the query end; those
records are filtered by `InstantRange` before merging.
### Risk Level
low
The change is limited to the physical high watermark used by MOR
incremental input splits. The logical query boundary remains unchanged.
Verification:
- `TestInputFormat#testReadArchivedCommitsIncrementallyForPreEightMOR`
passed.
- `TestIncrementalInputSplits#testInputSplitsForSplitLastCommit` passed.
- `TestIncrementalInputSplits` and `TestStreamReadMonitoringFunction`: 35
tests passed with no failures or errors.
- Checkstyle passed with 0 violations.
- `git diff --check` passed.
### Documentation Update
none. This is an internal correctness fix with no new user-facing behavior
or configuration.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
