rdblue commented on issue #25115: [SPARK-28351][SQL] Support DELETE in DataSource V2 URL: https://github.com/apache/spark/pull/25115#issuecomment-514802270 > We considered delete_by_filter and also delete_by_row, both have pros and cons. > if we want to provide a general DELETE support, or a future consideration of MERGE INTO or UPSERTS, delete via SupportOverwrite is not feasible Delete by expression is a much simpler case than row-level deletes, upserts, and merge into. Since the goal of this PR is to implement delete by expression, I suggest focusing on that so we can get it in. If you want to built the general solution for merge into, upsert, and row-level delete, that's a much longer design process. Since this doesn't require that process, let's separate the two. To do that, I think we should add `SupportsDelete` for filter-based deletes, or re-use `SupportsOverwrite`. > The table capabilities maybe a solution. My proposal was to use `SupportsOverwrite` to pass the filter and capabilities to prevent using that interface for overwrite if it isn't supported. I don't think that is the same thing as what you're talking about. > What do you think about the hybrid solution? Is that reasonable? I see no reason for a hybrid solution. Filter deletes are a simpler case and can be supported separately. When filters match expectations (e.g., partition filters for Hive, any filter for JDBC) then the source can use them. Otherwise filters can be rejected and Spark can fall back to row-level deletes, if those are supported. I don't see a reason to block filter-based deletes because those are not going to be the same thing as row-level deletes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
