rdblue commented on issue #25626: [SPARK-28892][SQL] Add UPDATE support for DataSource V2 URL: https://github.com/apache/spark/pull/25626#issuecomment-531430377 @xianyinxin, sorry for the late reply, I was at ApacheCon this week so I'm just catching up. My main concern is that this would cause source implementations that want to support `UPDATE` to call back into Spark from within the source to parallelize the update operation. That pattern is something we want to avoid: * Implementations that call into Spark would not share code (or, possibly, behavior) for a standard SQL operation. I would much rather work with people building these sources to get the implementation into Spark so that everyone can benefit, so that the code gets reviewed, and so that we can test thoroughly. * It creates SQL executions within SQL executions that are not tracked in the Spark UI and breaks `EXPLAIN`. This was a problem with the use of `RunnableCommand`, where metrics and physical plans were missing from the Spark UI. The SQL tab would only show a logical command name and `EXPLAIN` only had the logical plan. In this case, `EXPLAIN` would not show any detail about the physical operation for users to understand and tune. Adding the push-down API first doesn't necessarily mean that implementations would call into Spark like I'm concerned about. That's why I want to understand your use case. If you really need Spark to push operations to JDBC right away, then it may make sense to add this. But if you don't have an urgent need for it, I think Spark users would be better off designing and building the Spark-based implementation first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
