Yaoxuan Wu created SPARK-62140:
----------------------------------
Summary: Delta-based row-level UPDATE / MERGE skips rows whose
runtime filter attribute is NULL (merge-on-read update on a NULL partition does
nothing) since SPARK-56669
Key: SPARK-62140
URL: https://issues.apache.org/jira/browse/SPARK-62140
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Yaoxuan Wu
Since SPARK-56669 (4.2.0), a delta-based row-level UPDATE or MERGE skips
matching rows whose runtime filter attribute is NULL. A typical case is a
merge-on-read UPDATE on a partitioned Iceberg table: rows in the NULL partition
are not updated. The statement succeeds, and the table keeps the old values.
Group-based (copy-on-write) operations are correct, and so is the same
statement with
{{spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false}}.
{code:sql}
-- PySpark 4.2.0, iceberg-spark-runtime-4.2_2.13 1.13.0-SNAPSHOT, Iceberg SQL
extensions, hadoop catalog `ice`
CREATE TABLE ice.db.t (id BIGINT, p INT, v BOOLEAN) USING iceberg PARTITIONED
BY (p)
TBLPROPERTIES ('format-version'='2', 'write.update.mode'='merge-on-read',
'write.merge.mode'='merge-on-read');
INSERT INTO ice.db.t VALUES (1, NULL, true), (8, 5, true);
UPDATE ice.db.t SET v = false WHERE id = 1;
SELECT * FROM ice.db.t ORDER BY id;
-- expected: (1, NULL, false), (8, 5, true)
-- actual: (1, NULL, true), (8, 5, true) snapshot summary:
spark.update.num-updated-rows=0
-- same for
MERGE INTO ice.db.t t USING (SELECT 1 AS id) s ON t.id = s.id WHEN MATCHED THEN
UPDATE SET v = false;
{code}
* Correct with 'write.update.mode'='copy-on-write' /
'write.merge.mode'='copy-on-write'.
* Correct with {{SET
spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false}}.
* Same with {{PARTITIONED BY (bucket(4, p))}}, and with Parquet, ORC and Avro
data files.
* Correct on 4.1.1 (with iceberg-spark-runtime-4.1_2.13 1.12.0), where the rule
did not handle delta-based operations.
* The DELETE cases I tried were correct.
* The plan of the UPDATE shows the injected filter on the scan:
{{RuntimeFilters: [dynamicpruningexpression(p#17 IN subquery#149)]}}.
Cause:
* SPARK-56669 extended {{RowLevelOperationRuntimeGroupFiltering}} to
{{DeltaBasedRowLevelOperation}}. It filters the scan with
{{DynamicPruningExpression(InSubquery(filterAttributes, SELECT DISTINCT
filterAttributes FROM t WHERE cond))}}.
* For a group-based operation the attribute is a file name ({{_file}}), which
is never NULL. For a delta-based operation Iceberg declares its partition
source columns as the runtime filter attributes, and those can be NULL.
* {{p IN (subquery)}} is NULL, not true, when {{p}} is NULL, so the rows of the
NULL partition are pruned before the row-level operation sees them.
Possible fix: build a null-safe pruning condition for delta-based operations
(e.g. {{p IN (...) OR (p IS NULL AND <subquery has a NULL>)}}, or keep NULL
values in the pruning). Alternatively, skip group filtering on nullable filter
attributes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]