Hi Spark community: I would like to call a vote on SPIP Write schema narrowing for column-level UPDATE and MERGE in DSv2.
*Motivation* Today, Spark's row-level UPDATE and MERGE operations always read and write full rows, even when only a small subset of columns is modified. For wide tables common in AI/ML workloads, this leads to significant read and write amplification and prevents connectors from implementing efficient column-level update strategies. This SPIP proposes an opt-in Data Source V2 API that allows connectors to declare the exact data columns required for UPDATE and MERGE operations. Spark will narrow the write schema accordingly while preserving existing behavior for connectors that do not opt in. *Proposal (summary)* - Introduce a new SupportsColumnUpdates mix-in for row-level operations. - Add updatedColumns() to RowLevelOperationInfo so Spark can expose the columns being updated. - Add updateSchema() to LogicalWriteInfo to expose the narrowed write schema for update rows. - Add default DataWriter.writeUpdate(...) methods for handling narrowed update rows. - The design is fully additive and backward compatible. Existing connectors continue to receive full-row writes unless they explicitly opt into the new capability. Full API details, design rationale, rejected alternatives, and compatibility considerations are documented in the SPIP. *Relevant links* - SPIP <https://docs.google.com/document/d/1-Wiw9U54ESpbLakb9Cn_mO4AviM4nrk4TF7rNhI3JZg/edit?tab=t.0#heading=h.yoitjxhaitk8> - Discussion thread <https://www.mail-archive.com/dev%40spark.apache.org/msg35434.html> - JIRA <https://issues.apache.org/jira/browse/SPARK-56599> *Vote* Please vote on accepting this proposal as an official SPIP (the SPIP text above; implementation and follow-up JIRAs can land incrementally after acceptance). [ ] +1: Accept the proposal as an official SPIP [ ] +0: No opinion [ ] -1: I do not think we should adopt this SPIP (please explain why) The vote will remain open for at least 72 hours. Thanks to everyone who participated in the discussion and helped refine the design. Best, Huaxin
