[
https://issues.apache.org/jira/browse/SPARK-61384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61384:
-----------------------------------
Labels: pull-request-available (was: )
> Add experimental embedded Rust Spark Connect server with Spark JVM engine
> -------------------------------------------------------------------------
>
> Key: SPARK-61384
> URL: https://issues.apache.org/jira/browse/SPARK-61384
> Project: Spark
> Issue Type: New Feature
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Add an experimental, opt-in Spark Connect server frontend implemented in Rust
> and embedded in a Spark JVM through JNI.
> Rust owns the gRPC listener, request decoding, protocol sessions, operation
> IDs, response framing and stream lifetime. A direct JNI engine interface
> invokes SparkConnectPlanner/Catalyst and Spark execution APIs in the hosting
> JVM. The JVM is an execution backend; this does not forward service traffic
> to a second Spark Connect listener.
> Arrow vector value buffers cross JNI through the Arrow C Data Interface with
> explicit Java owner retention. Arrow IPC and gRPC still serialize results.
> The initial WIP proposes unpublished server/JNI crates, a Java host/engine
> adapter, runnable Rust and official PySpark examples, protocol and
> ownership/lifecycle tests, and build/acceptance documentation. Existing
> default client builds and client APIs stay unchanged.
> This is for trusted local workloads. RPC/query/data-type coverage is limited;
> authentication, authorization, TLS, full protocol parity and production
> resource governance are outside the prototype. The Java adapter uses internal
> Spark APIs and requires acceptance against a compatible assembly. Current
> local validation targets Spark 4.2.0-SNAPSHOT revision
> 250ccca3f2b5f043e20b415adf17bda5d202c60f, Scala 2.13 and JDK 21; the
> repository currently tracks protocol 4.3.0. Windows and JNI library unloading
> are unsupported.
> Generated-by: OpenAI Codex
> This pull request and its description were written by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]