fhan688 opened a new pull request, #3501:
URL: https://github.com/apache/fluss/pull/3501
### Purpose
Linked issue: https://github.com/apache/fluss/issues/3278
This PR completes the Hudi lake source read path by adding predicate
pushdown and sorted reader support on top of the existing Hudi split planner
and source reader work.
For Fluss primary-key tables backed by Hudi MOR tables, union read
requires the lake source reader to expose primary-key ordering. This PR adds a
Hudi sorted reader and wires Hudi filter pushdown into `HudiLakeSource`.
### Brief change log
- Add `FlussToHudiExpressionPredicateConverter` to convert Fluss
predicates to Hudi `ExpressionPredicates`.
- Supports `=`, `!=`, `<`, `<=`, `>`, `>=`, `IN`, `NOT IN`, `AND`, and
`OR`.
- Keeps unsupported predicates as remaining filters.
- Handles Hudi metadata-column offset when mapping Fluss field indexes
to Hudi schema fields.
- Builds binary `AND`/`OR` trees to match Hudi's predicate evaluation
behavior.
- Wire predicate pushdown into `HudiLakeSource`.
- Converts accepted filters in `withFilters`.
- Stores converted Hudi predicates and passes them to `HudiRecordReader`.
- Avoids loading Hudi schema for empty filter lists.
- Add `HudiSortedRecordReader` for Hudi primary-key/MOR tables.
- Delegates actual row reading to the existing `HudiRecordReader`.
- Exposes a primary-key comparator for Fluss union read.
- Supports empty lake split by returning an empty iterator while still
providing ordering.
- Derives primary-key fields from Hudi record key configuration.
- Add unit tests for:
- Hudi predicate conversion.
- Hudi sorted reader comparator behavior.
- Hudi source empty-filter behavior.
### Tests
- `mvn -pl fluss-lake/fluss-lake-hudi -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false
-Dtest=HudiLakeSourceTest,HudiSortedRecordReaderTest,FlussToHudiExpressionPredicateConverterTest
test`
- `mvn -pl fluss-lake/fluss-lake-hudi -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false test`
- `git diff --check`
### API and Format
No public API or storage format change.
### Documentation
No documentation change. This is an internal Hudi lake source
implementation improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
