fhan688 opened a new pull request, #3518:
URL: https://github.com/apache/fluss/pull/3518
### Purpose
Linked issue: https://github.com/apache/fluss/issues/3283
This PR completes the Hudi MOR tiering path by supporting Hudi compaction
in the lake tiering service.
Before this change, Hudi tiering could write and commit data files/log
files, but compaction write stats were not supported by the committer. As a
result, `table.datalake.auto-compaction` could not form a complete schedule ->
execute -> commit lifecycle for Hudi MOR tables.
### Brief change log
- Add `HudiCompactionService` to coordinate Hudi compaction scheduling,
inflight transition, bucket/partition-scoped execution,
commit, and rollback.
- Extend `HudiLakeWriter` to asynchronously execute inflight compaction
for MOR tables when data lake auto compaction is enabled.
- Extend `HudiLakeCommitter` to commit compaction results after normal
Hudi commit, schedule the next compaction round, and rollback compaction
instants on abort.
- Update Hudi source-side timeline handling to include completed
compaction instants, so committed compaction snapshots can be discovered and
read.
- Skip empty MOR file slices during split planning/verification to handle
pending or compacted file groups safely.
- Extend Hudi tiering tests to cover the full compaction lifecycle across
tiering rounds.
### Tests
- `mvn -pl fluss-lake/fluss-lake-hudi -Dcheckstyle.skip=true
spotless:apply`
- `git diff --check`
- `mvn -pl fluss-lake/fluss-lake-hudi -am -DskipITs=false
-Dcheckstyle.skip=true -DfailIfNoTests=false
-Dtest=HudiTieringTest,HudiTieringITCase test`
### API and Format
No public API changes.
No Fluss storage format changes.
This PR uses the existing data lake auto-compaction configuration path and
Hudi MOR compaction timeline semantics.
### Documentation
This introduces functional support for Hudi MOR auto compaction in tiering
service. No new configuration option is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
