I think we need wait for Herman to remove his VETO before we call a new vote. Herman do you have the cycles to look at this on Monday?
On Wed, Feb 25, 2026 at 9:59 PM vaquar khan <[email protected]> wrote: > > Hi everyone, > > I’ve gone through and addressed the comments from Herman and Xi Lyu in the > SPIP document. I believe the current proposal now covers the concerns > raised regarding the caching infrastructure for Phase 1. > > To make sure we don’t lose momentum on this: > > @Herman – Could you please take a final look at the doc when you have a > moment? > > Next Steps: If there are no further questions or concerns by EOD Sunday, > I’ll restart the formal vote this coming Monday. > > I’d like to get this wrapped up so we can start tackling these Connect > performance regressions. Thanks again to everyone for the push to get the > design right. > > SPIP Doc: > https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?pli=1&tab=t.0 > > Regards, > Vaquar Khan > > On Tue, 17 Feb 2026 at 21:59, vaquar khan <[email protected]> wrote: > >> Hi Herman, >> >> I have addressed all of your questions. Before I finalize the SPIP >> document with the feedback from the comments, I would appreciate it if you >> could review the comments and confirm that our proposal meets your >> expectations. >> >> SIP >> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?pli=1&tab=t.0 >> >> Regards, >> >> Vaquar Khan >> *Linkedin *-https://www.linkedin.com/in/vaquar-khan-b695577/ >> *Book *- >> https://us.amazon.com/stores/Vaquar-Khan/author/B0DMJCG9W6?ref=ap_rdr&shoppingPortalEnabled=true >> *GitBook*- >> https://vaquarkhan.github.io/microservices-recipes-a-free-gitbook/ >> *Stack *-https://stackoverflow.com/users/4812170/vaquar-khan >> *github*-https://github.com/vaquarkhan >> >> On Tue, 17 Feb 2026 at 14:08, Mich Talebzadeh <[email protected]> >> wrote: >> >>> @Herman van Hovell <[email protected]> >>> >>> good point. I added a comments in SPIP >>> >>> HTH >>> Dr Mich Talebzadeh, >>> Data Scientist | Distributed Systems (Spark) | Financial Forensics & >>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based >>> Analytics >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> >>> >>> >>> On Tue, 17 Feb 2026 at 19:18, Herman van Hovell via dev < >>> [email protected]> wrote: >>> >>>> Hi All, >>>> >>>> While I think it is great that we are trying to address this issue in >>>> Connect, I have concerns about the current proposal (see the comments in >>>> the doc). I would like to discuss this more in detail before proceeding. >>>> Given that this is an official vote, I will cast a -1 for now. >>>> >>>> Cheers, >>>> Herman >>>> >>>> On Tue, Feb 17, 2026 at 2:39 PM Devin Petersohn via dev < >>>> [email protected]> wrote: >>>> >>>>> +1 (non-binding). We've encountered the patterns described here >>>>> repeatedly in user workflows, and this proposal will be a big step forward >>>>> in the Spark Connect user experience. >>>>> >>>>> On Tue, Feb 17, 2026 at 12:07 PM Mich Talebzadeh < >>>>> [email protected]> wrote: >>>>> >>>>>> +1 from me >>>>>> >>>>>> Dr Mich Talebzadeh, >>>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics & >>>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based >>>>>> Analytics >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 17 Feb 2026 at 17:54, Holden Karau <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1, this fixes a key performance regression between regular Spark >>>>>>> and Spark connect. In talking with some users they ended up having to >>>>>>> implement their own caching to work around the death by 1k RPC issue >>>>>>> called >>>>>>> out here. >>>>>>> >>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> Pronouns: she/her >>>>>>> >>>>>>> >>>>>>> On Tue, Feb 17, 2026 at 8:28 AM vaquar khan <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Spark devs, >>>>>>>> >>>>>>>> I would like to call for a vote on the SPIP: Asynchronous Metadata >>>>>>>> Resolution & Lazy Prefetching for Spark Connect (Phase 1: Client-Side >>>>>>>> Plan-ID Caching). >>>>>>>> >>>>>>>> *Summary*: >>>>>>>> This proposal addresses the critical "Death by 1000 RPCs" >>>>>>>> performance regression in Spark Connect. Currently, interactive >>>>>>>> workloads >>>>>>>> suffer from blocking network latency during metadata resolution. The >>>>>>>> proposal introduces a Client-Side Plan-ID Cache to eliminate redundant >>>>>>>> RPCs >>>>>>>> for deterministic plan structures (e.g., select, withColumn), >>>>>>>> significantly >>>>>>>> improving interactive performance. >>>>>>>> >>>>>>>> *Scope*: >>>>>>>> Based on the discussion feedback (special thanks to Herman, Erik, >>>>>>>> Ruifeng, and Holden), this SPIP has been narrowed to Phase 1 only, >>>>>>>> focusing >>>>>>>> strictly on the caching infrastructure and excluding the broader >>>>>>>> asynchronous API changes for now. >>>>>>>> *Links*: >>>>>>>> >>>>>>>> *SPIP *Doc: >>>>>>>> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?usp=sharing >>>>>>>> >>>>>>>> *JIRA*: https://issues.apache.org/jira/browse/SPARK-55163 >>>>>>>> >>>>>>>> *Discussion Thread*: >>>>>>>> https://lists.apache.org/thread/wxj8mtopvm8bt959l58drzd4p90p6vn1 >>>>>>>> >>>>>>>> Please vote on the SPIP for the next 72 hours: >>>>>>>> >>>>>>>> [ ] +1: Accept the proposal as an official SPIP >>>>>>>> [ ] +0 >>>>>>>> [ ] -1: I don’t think this is a good idea because... >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Vaquar Khan >>>>>>>> *Linkedin *-https://www.linkedin.com/in/vaquar-khan-b695577/ >>>>>>>> *Book *- >>>>>>>> https://us.amazon.com/stores/Vaquar-Khan/author/B0DMJCG9W6?ref=ap_rdr&shoppingPortalEnabled=true >>>>>>>> *GitBook*- >>>>>>>> https://vaquarkhan.github.io/microservices-recipes-a-free-gitbook/ >>>>>>>> *Stack *-https://stackoverflow.com/users/4812170/vaquar-khan >>>>>>>> *github*-https://github.com/vaquarkhan >>>>>>>> >>>>>>> -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
