[ 
https://issues.apache.org/jira/browse/SPARK-62145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148300#comment-18148300
 ] 

Peter Toth commented on SPARK-62145:
------------------------------------

I'm working on it.

> A storage filter on the read of a group-based DELETE leaves rows out of 
> numDeletedRows
> --------------------------------------------------------------------------------------
>
>                 Key: SPARK-62145
>                 URL: https://issues.apache.org/jira/browse/SPARK-62145
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Peter Toth
>            Priority: Major
>
> A group-based DELETE takes {{numDeletedRows}} from the rows its read returns. 
> {{ReplaceDataExec.getDeleteSummary}} subtracts {{numCopiedRows}} from the 
> {{numOutputRows}} of that read's {{BatchScanExec}}. When the read gets a 
> runtime bloom filter as a storage filter, under 
> {{spark.sql.sources.v2.storageFilterPushdown.enabled}}, the rows its readers 
> reject never reach {{numOutputRows}}. So the summary leaves them out of the 
> deleted rows. The data the DELETE writes is still right.
> The read gets such a filter when the negated delete condition becomes a semi 
> join on the read of the rows to copy over, as in {{DELETE FROM t WHERE NOT 
> EXISTS (SELECT 1 FROM d WHERE d.id = t.id AND d.flag = 1)}}, with a scan that 
> implements {{SupportsStorageFilters}}.
> SPARK-62098 keeps the per-partition bloom filter from a group-based read. The 
> bloom filter of the whole creation side, which SPARK-60465 offers as a 
> storage filter, needs the same.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to