-1 (binding) for now.

Thank you for driving this, Holden. I support the goal. An OSS way to keep user 
code away from raw data is the missing half of SPARK-59726, and Spark Connect 
is a natural place for it. My -1 has two reasons.

First, the SPIP proposes a security boundary without defining it yet. 
SPARK-59726 v2 leaves isolating user code out of scope and names Spark Connect 
as the way to do it (G4, "Limit"), so this SPIP is where that boundary has to 
be written down.

Second, this vote combines two separate proposals: a Spark Connect data source 
with a trusted server mode for FGAC, and parallel reads from executors, a Spark 
Connect protocol change that "can benefit Spark Connect users in general". I'd 
like to vote on them separately.

I'll change my vote once group 1 is in the doc and parallel reads either move 
to their own SPIP or come with their protocol (groups 2 and 3).

### 1. Trust boundary (should be addressed before a vote)

- **What a client may send.** Today a Spark Connect client is not "just a 
normal user" (comment [e]). It can upload JARs and class files 
(`AddArtifacts`), send Scala UDFs as Java-serialized closures and Python 
UDFs/UDTFs/data sources, run shell commands with `SELECT TRANSFORM(...) USING`, 
call static Java methods such as `System.getenv` with `reflect` 
(`spark.sql.reflect.allowList` is empty by default), read files by path 
(`spark.read.load(path)`, ``SELECT * FROM parquet.`path` ``, `CREATE TEMPORARY 
VIEW ... USING`), and set `spark.sql.catalog.<name>` at runtime to open another 
catalog over the same storage. Each of these runs with the trusted cluster's 
credentials. Please list the allowed RPCs, plan nodes, functions and configs 
(default-deny), checked on the analyzed plan rather than on proto messages, 
since `SQL`, `expression_string` and `unresolved_function` can resolve to 
anything. `PipelinesHandler` (SQL command allowlist) and `MLHandler` (class 
allowlist) are precedents.
- **What the server sends back.** After each query, Spark Connect returns the 
SQL metrics of every plan node (`MetricGenerator`), including a scan's row 
count before the policy filter. `AnalyzePlan` returns `explain`, `tree_string` 
and `input_files`, which is the same data as `_file`. Please make sure 
SPARK-59726 G3 covers these responses, not only the UI and the event log.
- **Dependency on SPARK-59726.** Pushed-down operations run in the trusted 
optimizer next to the policy, so the boundary holds only if G1 (leakproof 
barrier) is enforced there. Please state the dependency and define "safe 
pushdown" as G1's leakproof. "The majority of the operations supported in DSv2 
are already safe" does not hold under ANSI mode: division, casts and overflow 
can fail on hidden rows, and the optimizer moves and combines filters freely 
(we fixed seven such cases this week alone, e.g. SPARK-62091).
- **Project policy.** Once Spark Connect is a security boundary, an allowlist 
bypass is a vulnerability for security@, and every new function, plan node, RPC 
and config needs a classification. Please say this explicitly.
- **Tests.** Please add a negative test for each path above to Q8.

### 2. Two proposals in one vote

- **Different dependencies and stakeholders.** The data source and the trusted 
mode build on SPARK-59726, while parallel reads build on the ADBC SPIP 
(SPARK-60255), which is still under discussion. Parallel reads also change what 
every Spark Connect client (Python, Scala, Go, Swift, Rust) and the Spark 
Connect Gateway see. With one vote, a voter cannot support one part and object 
to the other.
- **The scope of parallel reads is unclear.** The title and the vote email 
include them, but Q7 and Q8 have no milestone for them, Q6 says external 
storage may replace them, and the Future section lists their use by regular 
clients as future work. The DISCUSS thread on dev@ was titled "Spark Connect 
Data Source for Spark FGAC", and its emails did not mention them.
- **Suggestion.** Keep this SPIP to the data source and the trusted mode, 
moving data through the trusted driver or external storage first, and propose 
parallel reads as a separate SPIP with Appendix A (protocol and compatibility).

### 3. Parallel reads

- **Protocol.** "ADBC as a basis" leaves the wire protocol open. Spark Connect 
is not Flight SQL, and Spark has no Arrow Flight dependency today. Please 
specify it, for example a result mode that returns endpoints and tickets, which 
a client opts into through `RequestOption` like `ResultChunkingOptions`, so 
existing Python/Scala/Go/Swift/Rust clients are unaffected.
- **Serving from executors.** Task retries, speculation and stage retries 
re-read a partition, so results must stay until the reading job ends, and 
dynamic allocation must not remove an executor that holds unread results. 
Please define executor loss, cancellation and cleanup.
- **Security and networking.** Each trusted executor gets a new port that the 
untrusted cluster can reach. It needs TLS and short-lived tickets scoped to one 
operation, and it bypasses the Spark Connect Gateway. On Kubernetes, executor 
pods have no Service, and exposing them must not expose the driver RPC and 
block manager ports (`spark.authenticate` plus a network policy).
- **The non-parallel path** collects every partition on the trusted driver and 
is limited by `spark.driver.maxResultSize` (1g by default). Please state this 
limit.

### 4. Design and alternatives

- **TLS and identity.** The Spark Connect server cannot serve TLS itself; it 
listens in plaintext. The Python and Scala clients always use TLS when they 
send a token to a host other than `localhost`, so "SSL + shared token" needs a 
TLS-terminating proxy in front of the trusted server. In addition, 
`spark.connect.authenticate.token` is an internal config, and 
`UserContext.user_id` is declared by the client. With one shared token per 
cluster, each principal needs its own trusted cluster. The Spark Connect 
Gateway already verifies JWT/OIDC and replaces `user_id` with the verified 
identity, so it may be the better base for identity.
- **Version skew.** Without capability discovery, a client cannot know which 
pushdowns an older trusted server accepts, and a rejection fails the query. 
Please define the fallback.
- **DSv2 pushdown.** Join pushdown exists since 4.1.0 (SPARK-52187), and 
SPARK-60765, which you shepherd, adds set operations and window functions. 
Please name it in the doc.
- **JDBC.** The Thrift server is no longer needed: 4.1.0 ships a Spark Connect 
JDBC driver (`jdbc:sc://`, SPARK-53914). I tried the JDBC source and 
`JDBCTableCatalog` against a 4.3.0 Connect server, and both fail at 
`SparkConnectConnection.prepareStatement`, which is not implemented. With that 
and a dialect (4.3.0 can map `jdbc:sc:` to the built-in `databricks` dialect), 
the non-parallel path may need no new protocol. Worth comparing in Alternatives.

Thanks,
Dongjoon.



On 2026/11/20 18:19:21 huaxin gao wrote:
> +1
> 
> On Fri, Nov 20, 2026 at 9:50 AM Holden Karau <[email protected]> wrote:
> 
> > Hi Spark Community,
> >
> > I'm starting a vote on updating Spark to support reading from Spark
> > Connect, including parallel Spark Connect Reads. The original motivation is
> > to support fine grained access controls, but the parallel Spark Connect
> > reads can benefit Spark Connect users in general.
> >
> > SPIP doc:
> > https://docs.google.com/document/d/1c-Gw6HQn2akq7V2LWrvY8JYTo13CuPKHOa8gUT4QR7Y/edit?usp=sharing
> >
> > Only discussion happened in the doc (but the thread
> > https://lists.apache.org/thread.html/2ym2k633sd7kr1zsx86q9ccg6qn9h039 is
> > at for anyone who wants to verify)
> >
> > The vote will be open for at least 72 hours, but I plan to keep it open a
> > little longer, at least Wednesday @ 5pm pacific to give people time given
> > potential US holiday travel.
> >
> > Vote passes with 3 +1s from PMC and no -1s from PMC (
> > https://spark.apache.org/improvement-proposals.html)
> >
> > Cheers,
> >
> > Holden :)
> > --
> > Twitter: https://twitter.com/holdenkarau
> > Fight Health Insurance: https://www.fighthealthinsurance.com/
> > <https://www.fighthealthinsurance.com/?q=hk_email>
> > Books (Learning Spark, High Performance Spark, etc.):
> > https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> > Pronouns: she/her
> >
> 

---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to