fhan688 opened a new pull request, #19948:
URL: https://github.com/apache/hudi/pull/19948
### Describe the issue this Pull Request addresses
Part of #19286.
For pre-8 tables using a layout V1 timeline, Spark incremental queries can
use the 1.x reader with completion-time query boundaries. The query analyzer
converts these boundaries into a requested-time `InstantRange`.
When a MOR incremental query falls back to a full-table scan, the latest
file slice can contain log records beyond the requested end instant. The Spark
MOR file-group reader paths did not receive the analyzed `InstantRange`, so an
out-of-range update could participate in record merging and mask the latest
in-range version of the same key.
### Summary and Changelog
This PR propagates the incremental query's `InstantRange` to the MOR
file-group reader before record merging.
- Exposes the analyzed `InstantRange` from
`MergeOnReadIncrementalRelationV2`.
- Propagates the range through the default HadoopFsRelation/file-format
path.
- Propagates the same range through the legacy `HoodieMergeOnReadRDDV2`
path.
- Passes the range into the Spark reader context so base and log records
are filtered before merging.
- Keeps snapshot, V1 incremental, and other unaffected paths unchanged by
defaulting to an empty range.
- Adds regression coverage for V6 MOR tables read with the V8 incremental
reader, including archived instants, full-table-scan fallback, and an
out-of-range log update.
- Verifies both the default HadoopFsRelation path and the legacy RDD path.
### Impact
Fixes incremental read correctness for MOR pre-8 tables when the V8 reader
falls back to a full-table scan.
There are no storage format, configuration, or public API changes. Other
read paths continue to use an empty `InstantRange` and retain their existing
behavior.
### Risk Level
Low.
The change is limited to MOR incremental reads with an analyzed instant
range. Regression tests cover both Spark file-group reader entry paths and
confirm that a physically present log update after the query end instant is
excluded before merging.
Verification:
- `TestIncrementalReadWithFileGroupReader`: 9 tests passed, 0 failures, 0
errors.
- Relevant modules compile successfully with 0 scalastyle errors.
### Documentation Update
None. This is an internal correctness fix with no new user-facing
configuration or API.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
