gavin9402 opened a new pull request, #3677: URL: https://github.com/apache/celeborn/pull/3677
### What changes were proposed in this pull request? This PR introduces two major features to support **non-JVM (C++/Rust) clients** using Apache Celeborn for shuffle: **1. Standalone LifecycleManager Daemon (Scala/JVM)** - Added `LifecycleManagerDaemon` — a standalone JVM process that hosts a `LifecycleManager` independently from any compute engine (Spark/Flink) Driver. - Added `LifecycleManagerDaemonArguments` for CLI argument parsing (`--app-id`, `--master-endpoints`, `--port`, `--host`, `--properties-file`). - Added `sbin/start-lifecycle-manager.sh` launch script with classpath assembly, environment loading, and required-argument validation. - Updated `CelebornBuild.scala` and `service/pom.xml` to add `celeborn-client` as a dependency of the `service` module (needed because the Daemon instantiates `LifecycleManager` from the client module). **2. Rust SDK via C++ FFI (`rust/` directory)** - `celeborn-client-sys`: Low-level FFI crate using [cxx](https://cxx.rs/) to bridge Rust ↔ C++ Celeborn client. Includes: - `wrapper.h` / `wrapper.cc`: C++ shim exposing 7 functions (`create_client`, `setup_lifecycle_manager`, `shutdown`, `push_data`, `mapper_end`, `update_reducer_file_group`, `read_partition_full`). - `build.rs`: Build script linking Celeborn C++ static libs and system dependencies (folly, protobuf, abseil, boost, etc.) for macOS and Linux. - `celeborn-client`: Safe, ergonomic Rust wrapper providing `ShuffleClient` with: - Input validation (app_id non-empty, port > 0, codec ∈ {NONE, LZ4, ZSTD}). - `Drop`-safe shutdown (prevents double `ffi::shutdown` via `UniquePtr::null()` swap). - Convenience method `read_partition_all`. - Two example programs (`data_sum_writer.rs`, `data_sum_reader.rs`) mirroring the existing C++ `DataSumWithWriterClient` / `DataSumWithReaderClient` test programs. ### Why are the changes needed? Currently, `LifecycleManager` can only run **embedded inside a JVM-based compute engine Driver** (e.g., Spark Driver). This makes it impossible for non-JVM applications (Daft engine, etc.) to use Celeborn as their shuffle service, because: 1. The C++ client requires a running `LifecycleManager` to coordinate shuffle metadata (register shuffles, allocate slots, manage partition locations) with Celeborn Masters and Workers. 2. Without a standalone `LifecycleManager`, non-JVM applications have no way to bootstrap this coordination layer. By decoupling the `LifecycleManager` into a **standalone daemon process**, any client — regardless of language runtime — can connect to it via RPC. The Rust SDK then leverages this architecture to provide first-class Rust support by bridging to the existing, battle-tested C++ client implementation via FFI. ### Does this PR resolve a correctness bug? No ### Does this PR introduce _any_ user-facing change? Yes. - **New component**: Users can now start a standalone `LifecycleManager` daemon via `sbin/start-lifecycle-manager.sh --app-id <id> --master-endpoints <eps> --port <port>`. - **New SDK**: Rust applications can now use the `celeborn-client` crate to perform shuffle read/write operations against a Celeborn cluster. - **Limitation**: The standalone `LifecycleManager` does **not** support auth (`celeborn.auth.enabled` must be `false`), as the C++/Rust clients lack SASL support. ### How was this patch tested? - The Rust SDK was validated using the `data_sum_writer` and `data_sum_reader` example programs, which are Rust ports of the existing C++ integration tests (`DataSumWithWriterClient.cpp` / `DataSumWithReaderClient.cpp`). These write random numeric data across partitions and verify correctness by comparing partition sums between writer and reader. - The `LifecycleManagerDaemon` was tested by starting it against a local Celeborn cluster (Master + Workers) and verifying that the Rust examples can successfully connect, push data, and read data through the daemon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
