Hi Herman, I hope you are doing well. I am reaching out to you regarding the open VOTE thread for the "SPIP: Asynchronous Metadata Resolution & Lazy Prefetching (Phase 1)".
On March 10th, I provided a comprehensive update to the SPIP to directly address the technical concerns you raised with your -1 vote. Specifically, I detailed the distinction between the object-level cache and the structural-level cache, with examples to demonstrate why the structural plan cache is necessary for ecosystem tools. I have sent a few follow-ups since then but haven't heard back, and it has now been over two months since that technical discussion call happened. I deeply respect the veto process as a crucial tool for ensuring codebase integrity and triggering necessary consensus gathering. However, an extended pause leaves the development of this feature in a completely stalled state. Could you please let me know if the updates and examples provided on March 10th successfully addressed your architectural concerns? If they did, I would greatly appreciate it if you could formally withdraw your veto so the community can resume momentum on this initiative. If you still have outstanding technical concerns, I am fully available to discuss them and iterate further. Thank you for your time and your continued guidance on this proposal. Best regards, Vaquar Khan On Fri, 27 Feb 2026 at 16:43, Holden Karau <[email protected]> wrote: > I think we need wait for Herman to remove his VETO before we call a new > vote. Herman do you have the cycles to look at this on Monday? > > On Wed, Feb 25, 2026 at 9:59 PM vaquar khan <[email protected]> wrote: > >> >> Hi everyone, >> >> I’ve gone through and addressed the comments from Herman and Xi Lyu in >> the SPIP document. I believe the current proposal now covers the concerns >> raised regarding the caching infrastructure for Phase 1. >> >> To make sure we don’t lose momentum on this: >> >> @Herman – Could you please take a final look at the doc when you have a >> moment? >> >> Next Steps: If there are no further questions or concerns by EOD Sunday, >> I’ll restart the formal vote this coming Monday. >> >> I’d like to get this wrapped up so we can start tackling these Connect >> performance regressions. Thanks again to everyone for the push to get the >> design right. >> >> SPIP Doc: >> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?pli=1&tab=t.0 >> >> Regards, >> Vaquar Khan >> >> On Tue, 17 Feb 2026 at 21:59, vaquar khan <[email protected]> wrote: >> >>> Hi Herman, >>> >>> I have addressed all of your questions. Before I finalize the SPIP >>> document with the feedback from the comments, I would appreciate it if you >>> could review the comments and confirm that our proposal meets your >>> expectations. >>> >>> SIP >>> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?pli=1&tab=t.0 >>> >>> Regards, >>> >>> Vaquar Khan >>> *Linkedin *-https://www.linkedin.com/in/vaquar-khan-b695577/ >>> *Book *- >>> https://us.amazon.com/stores/Vaquar-Khan/author/B0DMJCG9W6?ref=ap_rdr&shoppingPortalEnabled=true >>> *GitBook*- >>> https://vaquarkhan.github.io/microservices-recipes-a-free-gitbook/ >>> *Stack *-https://stackoverflow.com/users/4812170/vaquar-khan >>> *github*-https://github.com/vaquarkhan >>> >>> On Tue, 17 Feb 2026 at 14:08, Mich Talebzadeh <[email protected]> >>> wrote: >>> >>>> @Herman van Hovell <[email protected]> >>>> >>>> good point. I added a comments in SPIP >>>> >>>> HTH >>>> Dr Mich Talebzadeh, >>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics & >>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based >>>> Analytics >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> >>>> >>>> >>>> On Tue, 17 Feb 2026 at 19:18, Herman van Hovell via dev < >>>> [email protected]> wrote: >>>> >>>>> Hi All, >>>>> >>>>> While I think it is great that we are trying to address this issue in >>>>> Connect, I have concerns about the current proposal (see the comments in >>>>> the doc). I would like to discuss this more in detail before proceeding. >>>>> Given that this is an official vote, I will cast a -1 for now. >>>>> >>>>> Cheers, >>>>> Herman >>>>> >>>>> On Tue, Feb 17, 2026 at 2:39 PM Devin Petersohn via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> +1 (non-binding). We've encountered the patterns described here >>>>>> repeatedly in user workflows, and this proposal will be a big step >>>>>> forward >>>>>> in the Spark Connect user experience. >>>>>> >>>>>> On Tue, Feb 17, 2026 at 12:07 PM Mich Talebzadeh < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +1 from me >>>>>>> >>>>>>> Dr Mich Talebzadeh, >>>>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics & >>>>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based >>>>>>> Analytics >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, 17 Feb 2026 at 17:54, Holden Karau <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> +1, this fixes a key performance regression between regular Spark >>>>>>>> and Spark connect. In talking with some users they ended up having to >>>>>>>> implement their own caching to work around the death by 1k RPC issue >>>>>>>> called >>>>>>>> out here. >>>>>>>> >>>>>>>> Twitter: https://twitter.com/holdenkarau >>>>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>>> Pronouns: she/her >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Feb 17, 2026 at 8:28 AM vaquar khan <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Spark devs, >>>>>>>>> >>>>>>>>> I would like to call for a vote on the SPIP: Asynchronous Metadata >>>>>>>>> Resolution & Lazy Prefetching for Spark Connect (Phase 1: Client-Side >>>>>>>>> Plan-ID Caching). >>>>>>>>> >>>>>>>>> *Summary*: >>>>>>>>> This proposal addresses the critical "Death by 1000 RPCs" >>>>>>>>> performance regression in Spark Connect. Currently, interactive >>>>>>>>> workloads >>>>>>>>> suffer from blocking network latency during metadata resolution. The >>>>>>>>> proposal introduces a Client-Side Plan-ID Cache to eliminate >>>>>>>>> redundant RPCs >>>>>>>>> for deterministic plan structures (e.g., select, withColumn), >>>>>>>>> significantly >>>>>>>>> improving interactive performance. >>>>>>>>> >>>>>>>>> *Scope*: >>>>>>>>> Based on the discussion feedback (special thanks to Herman, Erik, >>>>>>>>> Ruifeng, and Holden), this SPIP has been narrowed to Phase 1 only, >>>>>>>>> focusing >>>>>>>>> strictly on the caching infrastructure and excluding the broader >>>>>>>>> asynchronous API changes for now. >>>>>>>>> *Links*: >>>>>>>>> >>>>>>>>> *SPIP *Doc: >>>>>>>>> https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?usp=sharing >>>>>>>>> >>>>>>>>> *JIRA*: https://issues.apache.org/jira/browse/SPARK-55163 >>>>>>>>> >>>>>>>>> *Discussion Thread*: >>>>>>>>> https://lists.apache.org/thread/wxj8mtopvm8bt959l58drzd4p90p6vn1 >>>>>>>>> >>>>>>>>> Please vote on the SPIP for the next 72 hours: >>>>>>>>> >>>>>>>>> [ ] +1: Accept the proposal as an official SPIP >>>>>>>>> [ ] +0 >>>>>>>>> [ ] -1: I don’t think this is a good idea because... >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Vaquar Khan >>>>>>>>> *Linkedin *-https://www.linkedin.com/in/vaquar-khan-b695577/ >>>>>>>>> *Book *- >>>>>>>>> https://us.amazon.com/stores/Vaquar-Khan/author/B0DMJCG9W6?ref=ap_rdr&shoppingPortalEnabled=true >>>>>>>>> *GitBook*- >>>>>>>>> https://vaquarkhan.github.io/microservices-recipes-a-free-gitbook/ >>>>>>>>> *Stack *-https://stackoverflow.com/users/4812170/vaquar-khan >>>>>>>>> *github*-https://github.com/vaquarkhan >>>>>>>>> >>>>>>>> > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her >
