andygrove opened a new pull request, #7544: URL: https://github.com/apache/datafusion-comet/pull/7544
## Which issue does this PR close? Part of #7288. ## Rationale for this change Comet Native Executors run whole Spark queries on a cluster of `comet-scheduler` and `comet-executor` processes with no JVM in the data plane. The cluster is in `native/native-executors`. This PR adds the Spark driver side: the driver plans a query with Catalyst and Comet, turns the plan into a graph of stages, and submits it to the scheduler's Arrow Flight service. Spark tasks then read the result partitions from the executors over Flight. The driver never links Ballista. Its only new native dependencies are `arrow-flight` and `tonic`, and they are optional. The feature is off by default at every level: the `native-executors` Cargo feature is off, `spark.comet.nativeExecutors.enabled` is false, and both configs are internal. With the config off, `CometRule` behaves as before. This PR is split out of https://github.com/apache/datafusion-comet/pull/7289. The cluster-side features some queries need, such as sorting the partitions of a range exchange in parallel and moving a join whose build side turns out small, are in that PR and not here. Nothing here depends on Ballista's `comet-native` branch. The driver sends a few stage fields that the cluster in this repository ignores, and the docs page says which limitations come from that. ## What changes are included in this PR? One commit for each part. 1. The wire format and the JNI client. `native_executors.proto` gains `ResultEndpoints` and `ResultEndpoint` and the `output_ordering` and `estimated_output_bytes` fields of `Stage`. `native/core` gains the `native-executors` feature with optional `arrow-flight` and `tonic` dependencies, `native_executors/client.rs` (submit a query, open and read a result partition, close it), and `export_batch` becomes `pub(crate)` for the client. `native/Cargo.lock` is regenerated. 2. The Spark driver. `CometNativeExecutorsExec.scala` has the planner that builds the stage graph, refuses a query that cannot run entirely in the Comet pipeline (naming the operator), writes scalar subquery results into the plan as literals, and reads the result partitions. `NativeExecutorsClient.scala` has the JNI declarations and a clear error when libcomet lacks the feature. `CometRule` calls the planner when the config is on. `CometConf` gets `spark.comet.nativeExecutors.enabled` and `spark.comet.nativeExecutors.scheduler.url`. `CometNativeExecutorsSuite` is added, and `pom.xml` exempts the JNI declarations of `NativeExecutorsClient` from the unused-parameter check of the `strict-warnings` profile, as it does for `Native`. 3. Docs and CI. A contributor guide page, `native_executors.md`, with how to run the driver and its current limitations, and the one sentence of `native/native-executors/README.md` that said the driver was in progress. The CI parts are a clippy step and a nextest step for the `native-executors` feature in the Rust test action, and the manual suite's entry in `dev/ci/check-suites.py`. ## How are these changes tested? Rust, from `native/`: `cargo clippy -p datafusion-comet --all-targets --features native-executors -- -D warnings`, `cargo clippy --all-targets --workspace -- -D warnings`, `cargo fmt --check`, `cargo check -p datafusion-comet` (default build), and `cargo test -p datafusion-comet --features native-executors --lib native_executors`. The last runs the client's error-message tests. Scala, with JDK 17: `./mvnw spotless:check`, `./mvnw test-compile -DskipTests` (default profile and `-Pspark-3.5`), and `CometNativeExecutorsSuite`. Its two tests that fail while planning, for a scan that stays on the JVM and for dynamic partition pruning, pass. `CometExecSuite` passes with the config off. These runs used a libcomet built with the `native-executors` feature from the branch of #7289, not from this PR's Rust commit. The planning tests never reach the cluster, so they do not exercise the difference. Markdown passes `prettier --check`, and `dev/ci/check-suites.py` passes. The cluster-backed tests of `CometNativeExecutorsSuite` are manual, since they need the cluster binaries in `COMET_NATIVE_EXECUTORS_BIN_DIR`. They were not run for this PR against the cluster in this repository. The same suite passed against the cluster of #7289 at scale factor 10 before these changes, with 34 tests passing in both blocking and pipelined mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
