+1 (non-binding) Took a look, Thanks. Szehon
On Fri, Jul 10, 2026 at 2:18 PM vaquar khan <[email protected]> wrote: > +1 > Regards, > Viquar Khan > > On Fri, Jul 10, 2026, 4:05 PM Uroš Bojanić <[email protected]> wrote: > >> +1 (non-binding) >> >> This is a real pain for wide tables, thank you for working on it Huaxin! >> >> On 2026/07/07 22:41:28 huaxin gao wrote: >> > Hi Spark community: >> > >> > I would like to call a vote on SPIP Write schema narrowing for >> column-level >> > UPDATE and MERGE in DSv2. >> > >> > *Motivation* >> > >> > Today, Spark's row-level UPDATE and MERGE operations always read and >> write >> > full rows, even when only a small subset of columns is modified. For >> wide >> > tables common in AI/ML workloads, this leads to significant read and >> write >> > amplification and prevents connectors from implementing efficient >> > column-level update strategies. >> > >> > This SPIP proposes an opt-in Data Source V2 API that allows connectors >> to >> > declare the exact data columns required for UPDATE and MERGE operations. >> > Spark will narrow the write schema accordingly while preserving existing >> > behavior for connectors that do not opt in. >> > >> > *Proposal (summary)* >> > >> > >> > - Introduce a new SupportsColumnUpdates mix-in for row-level >> operations. >> > - Add updatedColumns() to RowLevelOperationInfo so Spark can expose >> the >> > columns being updated. >> > - Add updateSchema() to LogicalWriteInfo to expose the narrowed write >> > schema for update rows. >> > - Add default DataWriter.writeUpdate(...) methods for handling >> narrowed >> > update rows. >> > - The design is fully additive and backward compatible. Existing >> > connectors continue to receive full-row writes unless they >> explicitly opt >> > into the new capability. >> > >> > Full API details, design rationale, rejected alternatives, and >> > compatibility considerations are documented in the SPIP. >> > >> > *Relevant links* >> > >> > >> > - SPIP >> > < >> https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8 >> > >> > - Discussion thread >> > <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> >> > - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> >> > >> > *Vote* >> > >> > Please vote on accepting this proposal as an official SPIP (the SPIP >> text >> > above; implementation and follow-up JIRAs can land incrementally after >> > acceptance). >> > >> > [ ] +1: Accept the proposal as an official SPIP >> > >> > [ ] +0: No opinion >> > >> > [ ] -1: I do not think we should adopt this SPIP (please explain why) >> > >> > The vote will remain open for at least 72 hours. >> > >> > Thanks to everyone who participated in the discussion and helped refine >> the >> > design. >> > >> > Best, >> > >> > Huaxin >> > >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >>
