fhan688 opened a new pull request, #3456:
URL: https://github.com/apache/fluss/pull/3456
### Purpose
Linked issue: https://github.com/apache/fluss/issues/3276
This PR introduces the first planner-only Hudi lake source implementation
for Fluss.
Currently, the Hudi lake storage module supports basic Hudi catalog
integration, but
`HudiLakeStorage#createLakeSource` does not return a usable lake source.
As a result, Fluss cannot
plan readable lake splits for data already stored in Hudi.
This change adds Hudi source split planning based on a completed Hudi
instant. Record reading, limit
pushdown, and Hudi tiering writer support remain explicitly unsupported
and can be implemented in
follow-up PRs.
### Brief change log
- Add `HudiLakeSource`, `HudiSplit`, `HudiSplitPlanner`, and
`HudiSplitSerializer`.
- Wire `HudiLakeStorage#createLakeSource` to return a Hudi lake source.
- Add `HudiTableInfo` to resolve Hudi catalog table metadata, meta client,
completed timeline,
filesystem view, table type, partition fields, and bucket-aware metadata.
- Plan Hudi splits from the requested snapshot instant after validating
that the instant exists in
the completed Hudi timeline.
- Support split planning for:
- COW tables through latest base files before or on the requested
instant.
- MOR tables through latest merged file slices before or on the
requested instant.
- Persist Fluss bucket and partition metadata into Hudi table properties
for planner-side recovery.
- Return bucket `-1` for bucket-unaware Fluss log tables.
- Add and adjust UT coverage for:
- Hudi lake source planner/serializer wiring.
- Explicit unsupported limit and record reader behavior.
- Hudi split serialization version handling.
- Hudi split planner planning and missing instant behavior.
- Fluss bucket and partition metadata persisted in Hudi table properties.
- Partition value extraction from Hudi partition paths.
### Tests
- `mvn -pl fluss-lake/fluss-lake-hudi -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-Dtest=HudiLakeSourceTest,HudiSplitSerializerTest,HudiTableInfoTest,HudiConversionsTest,HudiSplitPlannerTest
test`
- `mvn -pl fluss-lake/fluss-lake-hudi -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test`
The Hudi module test run passed with 54 tests and 0 failures.
### API and Format
This PR does not change public Fluss APIs or Fluss storage format.
It adds internal Hudi table properties to preserve Fluss metadata for Hudi
source split planning:
- `fluss.bucket.keys`
- `fluss.bucket-aware`
- `fluss.partition.keys`
### Documentation
No user-facing documentation is added in this PR because this is a
planner-only foundation. Hudi
record reading and Hudi tiering writer support are still not exposed as
completed user workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
