ykourdi opened a new issue, #18162: URL: https://github.com/apache/iceberg/issues/18162
### Feature Request / Improvement Currently the row-level operation mode for `DELETE`/`UPDATE`/`MERGE` (copy-on-write vs merge-on-read) is resolved only from the `write.delete.mode` / `write.update.mode` / `write.merge.mode` table properties. There's no way to pick the mode for a single Spark session without mutating the table. ### Motivation / use case I run a periodic batch job that works off historical snapshots: it creates a branch from each one and rewrites rows there with a single `MERGE INTO` statement per historical snapshot. Merge-on-read is much cheaper for that work. In addition, I don't want to change how the main branch is written, other processes keep writing to it and should stay copy-on-write. The mode is a table property, so it's global. Flipping `write.merge.mode` to merge-on-read for my job flips it for those main branch writers too, so I'd have to lock the table for the duration to keep them off the transient mode. A session-scoped override only affects my job's writes and leaves everything else on the table's copy-on-write default. ### Proposal Add a session property that takes precedence over the `write.delete.mode` / `write.update.mode` / `write.merge.mode` table properties: - `spark.sql.iceberg.row-level-mode` (`copy-on-write` | `merge-on-read`) Precedence is session config > table property > default, so behavior is unchanged when it's unset. This follows `spark.sql.iceberg.distribution-mode`, which already overrides the per-command `write.*.distribution-mode` properties with one session key. The value is resolved through the normal config layering rather than reading the session conf directly: a `SparkWriteConf.rowLevelOperationMode(Command)` accessor goes through the existing `SparkConfParser` chain, and `SparkRowLevelOperationBuilder` calls it instead of reading `table.properties()`. ### Willingness to contribute Yes and I want to add support for Spark trees (`spark/v3.4`, `v3.5`, `v4.0`, `v4.1`) with tests + docs, and will open a PR if there's agreement on the approach. ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
