Casting my +1 as well. On Tue, Jul 7, 2026 at 4:22 PM Anton Okolnychyi <[email protected]> wrote:
> +1, this fits into the existing DML design in DSv2. > > - Anton > > вт, 7 лип. 2026 р. о 15:42 huaxin gao <[email protected]> пише: > >> Hi Spark community: >> >> I would like to call a vote on SPIP Write schema narrowing for >> column-level UPDATE and MERGE in DSv2. >> >> *Motivation* >> >> Today, Spark's row-level UPDATE and MERGE operations always read and >> write full rows, even when only a small subset of columns is modified. For >> wide tables common in AI/ML workloads, this leads to significant read and >> write amplification and prevents connectors from implementing efficient >> column-level update strategies. >> >> This SPIP proposes an opt-in Data Source V2 API that allows connectors to >> declare the exact data columns required for UPDATE and MERGE operations. >> Spark will narrow the write schema accordingly while preserving existing >> behavior for connectors that do not opt in. >> >> *Proposal (summary)* >> >> >> - Introduce a new SupportsColumnUpdates mix-in for row-level >> operations. >> - Add updatedColumns() to RowLevelOperationInfo so Spark can expose >> the columns being updated. >> - Add updateSchema() to LogicalWriteInfo to expose the narrowed write >> schema for update rows. >> - Add default DataWriter.writeUpdate(...) methods for handling >> narrowed update rows. >> - The design is fully additive and backward compatible. Existing >> connectors continue to receive full-row writes unless they explicitly opt >> into the new capability. >> >> Full API details, design rationale, rejected alternatives, and >> compatibility considerations are documented in the SPIP. >> >> *Relevant links* >> >> >> - SPIP >> >> <https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8> >> - Discussion thread >> <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> >> - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> >> >> *Vote* >> >> Please vote on accepting this proposal as an official SPIP (the SPIP text >> above; implementation and follow-up JIRAs can land incrementally after >> acceptance). >> >> [ ] +1: Accept the proposal as an official SPIP >> >> [ ] +0: No opinion >> >> [ ] -1: I do not think we should adopt this SPIP (please explain why) >> >> The vote will remain open for at least 72 hours. >> >> Thanks to everyone who participated in the discussion and helped refine >> the design. >> >> Best, >> >> Huaxin >> >
