DB Tsai created SPARK-61384:
-------------------------------

             Summary: Add experimental embedded Rust Spark Connect server with 
Spark JVM engine
                 Key: SPARK-61384
                 URL: https://issues.apache.org/jira/browse/SPARK-61384
             Project: Spark
          Issue Type: New Feature
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: DB Tsai


Add an experimental, opt-in Spark Connect server frontend implemented in Rust 
and embedded in a Spark JVM through JNI.

Rust owns the gRPC listener, request decoding, protocol sessions, operation 
IDs, response framing and stream lifetime. A direct JNI engine interface 
invokes SparkConnectPlanner/Catalyst and Spark execution APIs in the hosting 
JVM. The JVM is an execution backend; this does not forward service traffic to 
a second Spark Connect listener.

Arrow vector value buffers cross JNI through the Arrow C Data Interface with 
explicit Java owner retention. Arrow IPC and gRPC still serialize results.

The initial WIP proposes unpublished server/JNI crates, a Java host/engine 
adapter, runnable Rust and official PySpark examples, protocol and 
ownership/lifecycle tests, and build/acceptance documentation. Existing default 
client builds and client APIs stay unchanged.

This is for trusted local workloads. RPC/query/data-type coverage is limited; 
authentication, authorization, TLS, full protocol parity and production 
resource governance are outside the prototype. The Java adapter uses internal 
Spark APIs and requires acceptance against a compatible assembly. Current local 
validation targets Spark 4.2.0-SNAPSHOT revision 
250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21; the repository 
currently tracks protocol 4.3.0. Windows and JNI library unloading are 
unsupported.

Generated-by: OpenAI Codex
This pull request and its description were written by OpenAI Codex.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to