Peter Toth created SPARK-62145:
----------------------------------
Summary: A storage filter on the read of a group-based DELETE
leaves rows out of numDeletedRows
Key: SPARK-62145
URL: https://issues.apache.org/jira/browse/SPARK-62145
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
A group-based DELETE takes {{numDeletedRows}} from the rows its read returns.
{{ReplaceDataExec.getDeleteSummary}} subtracts {{numCopiedRows}} from the
{{numOutputRows}} of that read's {{BatchScanExec}}. When the read gets a
runtime bloom filter as a storage filter, under
{{spark.sql.sources.v2.storageFilterPushdown.enabled}}, the rows its readers
reject never reach {{numOutputRows}}. So the summary leaves them out of the
deleted rows. The data the DELETE writes is still right.
The read gets such a filter when the negated delete condition becomes a semi
join on the read of the rows to copy over, as in {{DELETE FROM t WHERE NOT
EXISTS (SELECT 1 FROM d WHERE d.id = t.id AND d.flag = 1)}}, with a scan that
implements {{SupportsStorageFilters}}.
SPARK-62098 keeps the per-partition bloom filter from a group-based read. The
bloom filter of the whole creation side, which SPARK-60465 offers as a storage
filter, needs the same.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]