Yaoxuan Wu created SPARK-62140:
----------------------------------

             Summary: Delta-based row-level UPDATE / MERGE skips rows whose 
runtime filter attribute is NULL (merge-on-read update on a NULL partition does 
nothing) since SPARK-56669
                 Key: SPARK-62140
                 URL: https://issues.apache.org/jira/browse/SPARK-62140
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Yaoxuan Wu


Since SPARK-56669 (4.2.0), a delta-based row-level UPDATE or MERGE skips 
matching rows whose runtime filter attribute is NULL. A typical case is a 
merge-on-read UPDATE on a partitioned Iceberg table: rows in the NULL partition 
are not updated. The statement succeeds, and the table keeps the old values. 
Group-based (copy-on-write) operations are correct, and so is the same 
statement with 
{{spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false}}.

{code:sql}
-- PySpark 4.2.0, iceberg-spark-runtime-4.2_2.13 1.13.0-SNAPSHOT, Iceberg SQL 
extensions, hadoop catalog `ice`
CREATE TABLE ice.db.t (id BIGINT, p INT, v BOOLEAN) USING iceberg PARTITIONED 
BY (p)
  TBLPROPERTIES ('format-version'='2', 'write.update.mode'='merge-on-read', 
'write.merge.mode'='merge-on-read');
INSERT INTO ice.db.t VALUES (1, NULL, true), (8, 5, true);

UPDATE ice.db.t SET v = false WHERE id = 1;
SELECT * FROM ice.db.t ORDER BY id;
-- expected: (1, NULL, false), (8, 5, true)
-- actual:   (1, NULL, true),  (8, 5, true)     snapshot summary: 
spark.update.num-updated-rows=0

-- same for
MERGE INTO ice.db.t t USING (SELECT 1 AS id) s ON t.id = s.id WHEN MATCHED THEN 
UPDATE SET v = false;
{code}

* Correct with 'write.update.mode'='copy-on-write' / 
'write.merge.mode'='copy-on-write'.
* Correct with {{SET 
spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled=false}}.
* Same with {{PARTITIONED BY (bucket(4, p))}}, and with Parquet, ORC and Avro 
data files.
* Correct on 4.1.1 (with iceberg-spark-runtime-4.1_2.13 1.12.0), where the rule 
did not handle delta-based operations.
* The DELETE cases I tried were correct.
* The plan of the UPDATE shows the injected filter on the scan: 
{{RuntimeFilters: [dynamicpruningexpression(p#17 IN subquery#149)]}}.

Cause:
* SPARK-56669 extended {{RowLevelOperationRuntimeGroupFiltering}} to 
{{DeltaBasedRowLevelOperation}}. It filters the scan with 
{{DynamicPruningExpression(InSubquery(filterAttributes, SELECT DISTINCT 
filterAttributes FROM t WHERE cond))}}.
* For a group-based operation the attribute is a file name ({{_file}}), which 
is never NULL. For a delta-based operation Iceberg declares its partition 
source columns as the runtime filter attributes, and those can be NULL.
* {{p IN (subquery)}} is NULL, not true, when {{p}} is NULL, so the rows of the 
NULL partition are pruned before the row-level operation sees them.

Possible fix: build a null-safe pruning condition for delta-based operations 
(e.g. {{p IN (...) OR (p IS NULL AND <subquery has a NULL>)}}, or keep NULL 
values in the pruning). Alternatively, skip group filtering on nullable filter 
attributes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to