[
https://issues.apache.org/jira/browse/SPARK-55163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093770#comment-18093770
]
Shekhar Prasad Rajak commented on SPARK-55163:
----------------------------------------------
Hi,
Are we having discussion on this over community call ? I would like to
contribute here.
> SPIP: Client-Side Metadata Caching for Spark Connect
> ----------------------------------------------------
>
> Key: SPARK-55163
> URL: https://issues.apache.org/jira/browse/SPARK-55163
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 3.4.0
> Reporter: vaquar khan
> Priority: Major
> Labels: connect, gsoc2026, mentor, pull-request-available, spark
>
> {panel}
> *This SPIP proposes adding a client-side schema cache for Spark Connect
> DataFrames.*
> Currently, every call to {{df.columns}} or {{df.schema}} triggers a
> synchronous gRPC analysis request to the server. While these are local and
> near-instant in Spark Classic, in Connect they average 277 ms on standard
> cloud setups (like AWS t3.medium). This makes iterative work extremely slow;
> we've measured a 13-second lag for 50 metadata calls in a typical ETL
> pipeline.
> This delay is forcing developers to use a "Shadow Schema" pattern, where they
> manually track column names in local lists to avoid the RPC overhead. Since
> Spark DataFrames are immutable, we can fix this by caching the resolved
> schema on the client after the first request. Our POC shows this reduces the
> 13-second lag to about 250 ms (a 51× speedup) without breaking the core Spark
> Connect model.
>
> I have followed the official SPIP template for the detailed breakdown below.
> *SIP*
> [
> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?tab=t.0|https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?tab=t.0]
> *Benchmark* -
> [https://docs.google.com/document/d/1ebX8CtTHN3Yf3AWxg7uttzaylxBLhEv-T94svhZg_uE/edit?tab=t.0]
> *Note for GSoC -*
> To set clear expectations for your GSoC timeline, and as a heads-up to the
> broader Spark developer community:
> Because the underlying SPIP (SPARK-55163) is still actively being discussed
> and has not yet received formal PMC approval, your GSoC project will function
> purely as an experimental prototype.
> Your open Pull Requests will be used by mentors to evaluate your GSoC
> deliverables and milestones. However, please be aware that your code will not
> be merged into the mainline Apache Spark repository during the GSoC program.
> Successfully completing your GSoC project and passing the evaluations is tied
> to the quality of your prototype and testing, not to getting the code merged.
> Your prototype will be incredibly valuable in helping the community benchmark
> the latency improvements for Spark Connect. I look forward to reviewing your
> finalized proposal!
> {panel}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
