-1 (binding) for now. Thank you for driving this, Holden. I support the goal. An OSS way to keep user code away from raw data is the missing half of SPARK-59726, and Spark Connect is a natural place for it. My -1 has two reasons.
First, the SPIP proposes a security boundary without defining it yet. SPARK-59726 v2 leaves isolating user code out of scope and names Spark Connect as the way to do it (G4, "Limit"), so this SPIP is where that boundary has to be written down. Second, this vote combines two separate proposals: a Spark Connect data source with a trusted server mode for FGAC, and parallel reads from executors, a Spark Connect protocol change that "can benefit Spark Connect users in general". I'd like to vote on them separately. I'll change my vote once group 1 is in the doc and parallel reads either move to their own SPIP or come with their protocol (groups 2 and 3). ### 1. Trust boundary (should be addressed before a vote) - **What a client may send.** Today a Spark Connect client is not "just a normal user" (comment [e]). It can upload JARs and class files (`AddArtifacts`), send Scala UDFs as Java-serialized closures and Python UDFs/UDTFs/data sources, run shell commands with `SELECT TRANSFORM(...) USING`, call static Java methods such as `System.getenv` with `reflect` (`spark.sql.reflect.allowList` is empty by default), read files by path (`spark.read.load(path)`, ``SELECT * FROM parquet.`path` ``, `CREATE TEMPORARY VIEW ... USING`), and set `spark.sql.catalog.<name>` at runtime to open another catalog over the same storage. Each of these runs with the trusted cluster's credentials. Please list the allowed RPCs, plan nodes, functions and configs (default-deny), checked on the analyzed plan rather than on proto messages, since `SQL`, `expression_string` and `unresolved_function` can resolve to anything. `PipelinesHandler` (SQL command allowlist) and `MLHandler` (class allowlist) are precedents. - **What the server sends back.** After each query, Spark Connect returns the SQL metrics of every plan node (`MetricGenerator`), including a scan's row count before the policy filter. `AnalyzePlan` returns `explain`, `tree_string` and `input_files`, which is the same data as `_file`. Please make sure SPARK-59726 G3 covers these responses, not only the UI and the event log. - **Dependency on SPARK-59726.** Pushed-down operations run in the trusted optimizer next to the policy, so the boundary holds only if G1 (leakproof barrier) is enforced there. Please state the dependency and define "safe pushdown" as G1's leakproof. "The majority of the operations supported in DSv2 are already safe" does not hold under ANSI mode: division, casts and overflow can fail on hidden rows, and the optimizer moves and combines filters freely (we fixed seven such cases this week alone, e.g. SPARK-62091). - **Project policy.** Once Spark Connect is a security boundary, an allowlist bypass is a vulnerability for security@, and every new function, plan node, RPC and config needs a classification. Please say this explicitly. - **Tests.** Please add a negative test for each path above to Q8. ### 2. Two proposals in one vote - **Different dependencies and stakeholders.** The data source and the trusted mode build on SPARK-59726, while parallel reads build on the ADBC SPIP (SPARK-60255), which is still under discussion. Parallel reads also change what every Spark Connect client (Python, Scala, Go, Swift, Rust) and the Spark Connect Gateway see. With one vote, a voter cannot support one part and object to the other. - **The scope of parallel reads is unclear.** The title and the vote email include them, but Q7 and Q8 have no milestone for them, Q6 says external storage may replace them, and the Future section lists their use by regular clients as future work. The DISCUSS thread on dev@ was titled "Spark Connect Data Source for Spark FGAC", and its emails did not mention them. - **Suggestion.** Keep this SPIP to the data source and the trusted mode, moving data through the trusted driver or external storage first, and propose parallel reads as a separate SPIP with Appendix A (protocol and compatibility). ### 3. Parallel reads - **Protocol.** "ADBC as a basis" leaves the wire protocol open. Spark Connect is not Flight SQL, and Spark has no Arrow Flight dependency today. Please specify it, for example a result mode that returns endpoints and tickets, which a client opts into through `RequestOption` like `ResultChunkingOptions`, so existing Python/Scala/Go/Swift/Rust clients are unaffected. - **Serving from executors.** Task retries, speculation and stage retries re-read a partition, so results must stay until the reading job ends, and dynamic allocation must not remove an executor that holds unread results. Please define executor loss, cancellation and cleanup. - **Security and networking.** Each trusted executor gets a new port that the untrusted cluster can reach. It needs TLS and short-lived tickets scoped to one operation, and it bypasses the Spark Connect Gateway. On Kubernetes, executor pods have no Service, and exposing them must not expose the driver RPC and block manager ports (`spark.authenticate` plus a network policy). - **The non-parallel path** collects every partition on the trusted driver and is limited by `spark.driver.maxResultSize` (1g by default). Please state this limit. ### 4. Design and alternatives - **TLS and identity.** The Spark Connect server cannot serve TLS itself; it listens in plaintext. The Python and Scala clients always use TLS when they send a token to a host other than `localhost`, so "SSL + shared token" needs a TLS-terminating proxy in front of the trusted server. In addition, `spark.connect.authenticate.token` is an internal config, and `UserContext.user_id` is declared by the client. With one shared token per cluster, each principal needs its own trusted cluster. The Spark Connect Gateway already verifies JWT/OIDC and replaces `user_id` with the verified identity, so it may be the better base for identity. - **Version skew.** Without capability discovery, a client cannot know which pushdowns an older trusted server accepts, and a rejection fails the query. Please define the fallback. - **DSv2 pushdown.** Join pushdown exists since 4.1.0 (SPARK-52187), and SPARK-60765, which you shepherd, adds set operations and window functions. Please name it in the doc. - **JDBC.** The Thrift server is no longer needed: 4.1.0 ships a Spark Connect JDBC driver (`jdbc:sc://`, SPARK-53914). I tried the JDBC source and `JDBCTableCatalog` against a 4.3.0 Connect server, and both fail at `SparkConnectConnection.prepareStatement`, which is not implemented. With that and a dialect (4.3.0 can map `jdbc:sc:` to the built-in `databricks` dialect), the non-parallel path may need no new protocol. Worth comparing in Alternatives. Thanks, Dongjoon. On 2026/11/20 18:19:21 huaxin gao wrote: > +1 > > On Fri, Nov 20, 2026 at 9:50 AM Holden Karau <[email protected]> wrote: > > > Hi Spark Community, > > > > I'm starting a vote on updating Spark to support reading from Spark > > Connect, including parallel Spark Connect Reads. The original motivation is > > to support fine grained access controls, but the parallel Spark Connect > > reads can benefit Spark Connect users in general. > > > > SPIP doc: > > https://docs.google.com/document/d/1c-Gw6HQn2akq7V2LWrvY8JYTo13CuPKHOa8gUT4QR7Y/edit?usp=sharing > > > > Only discussion happened in the doc (but the thread > > https://lists.apache.org/thread.html/2ym2k633sd7kr1zsx86q9ccg6qn9h039 is > > at for anyone who wants to verify) > > > > The vote will be open for at least 72 hours, but I plan to keep it open a > > little longer, at least Wednesday @ 5pm pacific to give people time given > > potential US holiday travel. > > > > Vote passes with 3 +1s from PMC and no -1s from PMC ( > > https://spark.apache.org/improvement-proposals.html) > > > > Cheers, > > > > Holden :) > > -- > > Twitter: https://twitter.com/holdenkarau > > Fight Health Insurance: https://www.fighthealthinsurance.com/ > > <https://www.fighthealthinsurance.com/?q=hk_email> > > Books (Learning Spark, High Performance Spark, etc.): > > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > Pronouns: she/her > > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
