fhan688 opened a new pull request, #3507:
URL: https://github.com/apache/fluss/pull/3507
### Purpose
Linked issue: https://github.com/apache/fluss/issues/3280
This PR introduces the writer-side implementation for tiering Fluss tables
to Hudi.
It builds on the previous lake/hudi PRs that added Hudi catalog/source
support and the tiering writer init metadata such as split index and tiering
round timestamp. This PR only migrates the Hudi writer capability. The Hudi
committer and full `LakeStorage#createLakeTieringFactory` enablement will be
completed in follow-up PRs.
### Brief change log
- Add Hudi lake tiering writer scaffolding:
- `HudiLakeTieringFactory`
- `HudiLakeWriter`
- `HudiWriteResult`
- `HudiWriteResultSerializer`
- `HudiWriteTableInfo`
- Add writer-side record conversion and buffering:
- Convert Fluss `LogRecord` / `InternalRow` into Hudi-compatible Flink
`RowData`.
- Map Fluss `ChangeType` to Flink `RowKind`.
- Add Hudi bucket/file-id routing for insert/upsert writes.
- Buffer and flush records through `HoodieFlinkWriteClient`.
- Add Hudi instant coordination metadata:
- Introduce DFS-backed checkpoint metadata for sharing the initialized
Hudi instant across tiering writer subtasks.
- Use `WriterInitContext#splitIndex()` to let only the first split
initialize the Hudi instant.
- Use `WriterInitContext#tieringRoundTimestamp()` to wait for the
correct round instant.
- Keep the feature intentionally writer-only for now:
- Hudi committer APIs still throw `UnsupportedOperationException`.
- `HudiLakeStorage#createLakeTieringFactory()` is not enabled yet to
avoid exposing incomplete tiering-to-Hudi service behavior.
- Add `flink-table-runtime` as a provided dependency for the Hudi writer
buffer.
### Tests
- `mvn -q -pl fluss-lake/fluss-lake-hudi -am -DskipITs
-Dcheckstyle.skip=true -DfailIfNoTests=false
-Dtest=FlussRecordAsHudiRowTest,HudiWriteResultSerializerTest test`
- `git diff --check`
`mvn clean verify` was not run locally.
### API and Format
This PR does not introduce a new user-facing public API.
It adds internal Hudi tiering writer classes and a versioned
`HudiWriteResultSerializer`. It also introduces internal checkpoint metadata
files under Hudi auxiliary metadata paths for coordinating writer-side instant
initialization. The full committable format and commit protocol will be
finalized in the follow-up committer PR.
### Documentation
No user-facing documentation is added in this PR because Hudi tiering is
not fully enabled yet. Documentation should be added when the committer and
end-to-end Hudi tiering service are enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
