+1 On 2026/07/08 07:22:48 Peter Toth wrote: > +1 > > On Wed 8. Jul 2026 at 01:53, huaxin gao <[email protected]> wrote: > > > Casting my +1 as well. > > > > On Tue, Jul 7, 2026 at 4:22 PM Anton Okolnychyi <[email protected]> > > wrote: > > > >> +1, this fits into the existing DML design in DSv2. > >> > >> - Anton > >> > >> вт, 7 лип. 2026 р. о 15:42 huaxin gao <[email protected]> пише: > >> > >>> Hi Spark community: > >>> > >>> I would like to call a vote on SPIP Write schema narrowing for > >>> column-level UPDATE and MERGE in DSv2. > >>> > >>> *Motivation* > >>> > >>> Today, Spark's row-level UPDATE and MERGE operations always read and > >>> write full rows, even when only a small subset of columns is modified. For > >>> wide tables common in AI/ML workloads, this leads to significant read and > >>> write amplification and prevents connectors from implementing efficient > >>> column-level update strategies. > >>> > >>> This SPIP proposes an opt-in Data Source V2 API that allows connectors > >>> to declare the exact data columns required for UPDATE and MERGE > >>> operations. > >>> Spark will narrow the write schema accordingly while preserving existing > >>> behavior for connectors that do not opt in. > >>> > >>> *Proposal (summary)* > >>> > >>> > >>> - Introduce a new SupportsColumnUpdates mix-in for row-level > >>> operations. > >>> - Add updatedColumns() to RowLevelOperationInfo so Spark can expose > >>> the columns being updated. > >>> - Add updateSchema() to LogicalWriteInfo to expose the narrowed > >>> write schema for update rows. > >>> - Add default DataWriter.writeUpdate(...) methods for handling > >>> narrowed update rows. > >>> - The design is fully additive and backward compatible. Existing > >>> connectors continue to receive full-row writes unless they explicitly > >>> opt > >>> into the new capability. > >>> > >>> Full API details, design rationale, rejected alternatives, and > >>> compatibility considerations are documented in the SPIP. > >>> > >>> *Relevant links* > >>> > >>> > >>> - SPIP > >>> > >>> <https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8> > >>> - Discussion thread > >>> <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> > >>> - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> > >>> > >>> *Vote* > >>> > >>> Please vote on accepting this proposal as an official SPIP (the SPIP > >>> text above; implementation and follow-up JIRAs can land incrementally > >>> after > >>> acceptance). > >>> > >>> [ ] +1: Accept the proposal as an official SPIP > >>> > >>> [ ] +0: No opinion > >>> > >>> [ ] -1: I do not think we should adopt this SPIP (please explain why) > >>> > >>> The vote will remain open for at least 72 hours. > >>> > >>> Thanks to everyone who participated in the discussion and helped refine > >>> the design. > >>> > >>> Best, > >>> > >>> Huaxin > >>> > >> >
--------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
