Re: [DISCUSS][Spark SQL] Update API

2024-09-24 Thread Wenchen Fan
All the existing DML APIs we support today have a source query so they all start with the source DataFrame, e.g. sourceDf.write.insertInto... sourceDf.write.saveAsTable... sourceDf.mergeInto... However, this is not the case for UPDATE and DELETE, as there is no source query. We need a different st

[DISCUSS][Spark SQL] Update API

2024-09-23 Thread Szehon Ho
Hi all, In https://github.com/apache/spark/pull/47233, we are looking to add a Spark DataFrame API for functional equivalence to Spark SQL's UPDATE statement. There are open discussions on the PR about location/format of the API, and we wanted to ask on devlist to get more opinions. One consider