cloud-fan commented on pull request #35395: URL: https://github.com/apache/spark/pull/35395#issuecomment-1067180811
@aokolnychyi thanks for your patience to resolve my concerns! IIUC, the overall process of group-based DELETE is: 1. Build a SQL query to calculate the groups that need to be replaced. The query scans the table, then filter by the DELETE condition, and finally select a hidden column that represents group id. This query can leverage the exiting optimizer framework to do filter pushdown, or even runtime filter pushdown if the DELETE condition has subqueries and this query is planned as a join. 2. The data source writer gets the collected group ids, scan these groups, then filter by the opposite of the DELETE condition to get the data after deletion, and finally replace these groups. Note that, the above process scans some groups twice to get the minimum affected groups. We can also have a simplified process to only scan once but may overestimate the groups to be replaced: 1. push down the DELETE condition to the source, so that it can estimate the groups that need to be replaced 2. The data source writer gets the collected group ids, scan these groups, then filter by the opposite of the DELETE condition to get the data after deletion, and finally replace these groups. The major difference is, instead of running a SQL query to collect affected group ids, we can use the static filter from the DELETE condition to estimate the affected groups, which does not need to scan these groups. Now let's look at the proposed API. I think it works pretty well for the simplified process, and I believe you have a plan to support the non-simple process by adding extension APIs. Just to make sure it will be backward compatible and we are on the same page: 1. We will add a subclass of `SupportsRowLevelOperaton` 2. The subclass has a new method to return a `ScanBuilder` which is used to collect the affected groups accurately and has a hidden column to represent group id. 3. `SupportsRowLevelOperaton.newScanBuilder` should return a `ScanBuilder` implementing `SupportsRuntimeFilter`, so that it can get the affected groups as an IN predicate. Other questions about this PR: 1. row-level operation does not even call out the data writing behavior. Shall we consider other names like `SupportsDataUpdating`, `SupportsUPSERT`, et.? 2. Why do we put the replace plan as a parameter of `DeleteFromTable`? Can we simply replace `DeleteFromTable` with the new replace plan in the rule? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
