SteNicholas opened a new pull request, #3751: URL: https://github.com/apache/celeborn/pull/3751
### What changes were proposed in this pull request? This PR adds an Apache Flink 2.3 client for Celeborn: - Add the `client-flink/flink-2.3` and `client-flink/flink-2.3-shaded` modules. - Introduce `CelebornFlinkShimV23` and `tiered/CelebornTierFactory` for Flink 2.3, registered via the `org.apache.celeborn.plugin.flink.CelebornFlinkShim` service file. - Adapt the common `RemoteShuffleInputGate` to the Flink 2.3 `IndexedInputGate`/`InputGate` API surface: add `getBufferFilteringCompleteFuture()` and the `setCheckpointingDuringRecoveryEnabled`/`isCheckpointingDuringRecoveryEnabled` hooks, and pass the extra recovered-state argument (`new ArrayDeque<>()`) introduced by the recovery-checkpointing work in Flink 2.3. - Port the existing Flink client UT suites to the new module (`RemoteShuffleMasterSuiteJV23`, `RemoteShuffleResultPartitionSuiteJV23`, `RemoteShuffleResultPartitionFactorySuiteJV23`, `RemoteShuffleServiceFactorySuiteJV23`, `ShuffleResourceTrackerSuiteJV23`, `tiered/CelebornTierMasterAgentSuiteJV23`). - Wire the new module into the build/release/CI: `pom.xml`, `project/CelebornBuild.scala`, `build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`, the `deps`/`license`/`style` workflows, and add the `dev/deps/dependencies-client-flink-2.3` dependency list plus the shaded `LICENSE`/`NOTICE` files. - Update docs (`README.md`, `docs/README.md`, `docs/deploy.md`, `docs/developers/overview.md`, `docs/developers/sbt.md`) to include Flink 2.3. ### Why are the changes needed? [Apache Flink 2.3.0](https://flink.apache.org/2026/06/25/apache-flink-2.3.0-release-announcement/) was released on June 25, 2026. Among other things it adds support for checkpointing during unaligned-checkpoint recovery, which changed the `InputGate` API that the Celeborn Flink client integrates against: - [FLINK-38543] [apache/flink#27862](https://github.com/apache/flink/pull/27862) — change the overall UC restore process, JM and task initialization (introduces `bufferFilteringCompleteFuture` and the checkpointing-during-recovery transition). - [FLINK-38930] [apache/flink#27783](https://github.com/apache/flink/pull/27783) — filter records before processing without a spilling strategy during recovery. - [FLINK-39018] [apache/flink#27861](https://github.com/apache/flink/pull/27861) — support checkpoint for `LocalInputChannel`. Celeborn needs a dedicated Flink 2.3 client so users running on Flink 2.3 can use remote shuffle. ### Does this PR resolve a correctness bug? - [ ] Yes No. ### Does this PR introduce _any_ user-facing change? - [x] Yes A new Flink 2.3 client is available via the `-Pflink-2.3` profile, the release binary now ships a `celeborn-client-flink-2.3-shaded` jar, and the docs list Flink 2.3 as a supported version. ### How was this patch tested? The existing Flink client UT suites are ported to the new `flink-2.3` module and run under the `-Pflink-2.3` profile. The `deps`, `license`, and `style` CI workflows are extended to cover `flink-2.3`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
