[
https://issues.apache.org/jira/browse/SPARK-62145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148300#comment-18148300
]
Peter Toth commented on SPARK-62145:
------------------------------------
I'm working on it.
> A storage filter on the read of a group-based DELETE leaves rows out of
> numDeletedRows
> --------------------------------------------------------------------------------------
>
> Key: SPARK-62145
> URL: https://issues.apache.org/jira/browse/SPARK-62145
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> A group-based DELETE takes {{numDeletedRows}} from the rows its read returns.
> {{ReplaceDataExec.getDeleteSummary}} subtracts {{numCopiedRows}} from the
> {{numOutputRows}} of that read's {{BatchScanExec}}. When the read gets a
> runtime bloom filter as a storage filter, under
> {{spark.sql.sources.v2.storageFilterPushdown.enabled}}, the rows its readers
> reject never reach {{numOutputRows}}. So the summary leaves them out of the
> deleted rows. The data the DELETE writes is still right.
> The read gets such a filter when the negated delete condition becomes a semi
> join on the read of the rows to copy over, as in {{DELETE FROM t WHERE NOT
> EXISTS (SELECT 1 FROM d WHERE d.id = t.id AND d.flag = 1)}}, with a scan that
> implements {{SupportsStorageFilters}}.
> SPARK-62098 keeps the per-partition bloom filter from a group-based read. The
> bloom filter of the whole creation side, which SPARK-60465 offers as a
> storage filter, needs the same.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]