[ 
https://issues.apache.org/jira/browse/SPARK-61384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DB Tsai updated SPARK-61384:
----------------------------
    Fix Version/s: connect-rust-4.3.0

> Add experimental embedded Rust Spark Connect server with Spark JVM engine
> -------------------------------------------------------------------------
>
>                 Key: SPARK-61384
>                 URL: https://issues.apache.org/jira/browse/SPARK-61384
>             Project: Spark
>          Issue Type: New Feature
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: DB Tsai
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> Add an experimental, opt-in Spark Connect server frontend implemented in Rust 
> and embedded in a Spark JVM through JNI.
> Rust owns the gRPC listener, request decoding, protocol sessions, operation 
> IDs, response framing and stream lifetime. A direct JNI engine interface 
> invokes SparkConnectPlanner/Catalyst and Spark execution APIs in the hosting 
> JVM. The JVM is an execution backend; this does not forward service traffic 
> to a second Spark Connect listener.
> Arrow vector value buffers cross JNI through the Arrow C Data Interface with 
> explicit Java owner retention. Arrow IPC and gRPC still serialize results.
> The initial WIP proposes unpublished server/JNI crates, a Java host/engine 
> adapter, runnable Rust and official PySpark examples, protocol and 
> ownership/lifecycle tests, and build/acceptance documentation. Existing 
> default client builds and client APIs stay unchanged.
> This is for trusted local workloads. RPC/query/data-type coverage is limited; 
> authentication, authorization, TLS, full protocol parity and production 
> resource governance are outside the prototype. The Java adapter uses internal 
> Spark APIs and requires acceptance against a compatible assembly. Current 
> local validation targets Spark 4.2.0-SNAPSHOT revision 
> 250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21; the 
> repository currently tracks protocol 4.3.0. Windows and JNI library unloading 
> are unsupported.
> Generated-by: OpenAI Codex
> This pull request and its description were written by OpenAI Codex.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to