Peter Toth created SPARK-62145:
----------------------------------

             Summary: A storage filter on the read of a group-based DELETE 
leaves rows out of numDeletedRows
                 Key: SPARK-62145
                 URL: https://issues.apache.org/jira/browse/SPARK-62145
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Peter Toth


A group-based DELETE takes {{numDeletedRows}} from the rows its read returns. 
{{ReplaceDataExec.getDeleteSummary}} subtracts {{numCopiedRows}} from the 
{{numOutputRows}} of that read's {{BatchScanExec}}. When the read gets a 
runtime bloom filter as a storage filter, under 
{{spark.sql.sources.v2.storageFilterPushdown.enabled}}, the rows its readers 
reject never reach {{numOutputRows}}. So the summary leaves them out of the 
deleted rows. The data the DELETE writes is still right.

The read gets such a filter when the negated delete condition becomes a semi 
join on the read of the rows to copy over, as in {{DELETE FROM t WHERE NOT 
EXISTS (SELECT 1 FROM d WHERE d.id = t.id AND d.flag = 1)}}, with a scan that 
implements {{SupportsStorageFilters}}.

SPARK-62098 keeps the per-partition bloom filter from a group-based read. The 
bloom filter of the whole creation side, which SPARK-60465 offers as a storage 
filter, needs the same.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to