This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 8cb0e6ef5ef4 [SPARK-60849][SQL] Fix wrong results by ConstantFolding 
for size of a map with duplicate keys
8cb0e6ef5ef4 is described below

commit 8cb0e6ef5ef40142eb7a35154e59168249eae278
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 16:57:50 2026 -0800

    [SPARK-60849][SQL] Fix wrong results by ConstantFolding for size of a map 
with duplicate keys
    
    ### What changes were proposed in this pull request?
    
    This PR fixes a correctness issue where `ConstantFolding` folds 
`size(map(...))` and `cardinality(map(...))` to the number of key-value pairs, 
although building the map removes duplicate keys. With 
`spark.sql.mapKeyDedupPolicy=LAST_WIN`, the query returns a wrong size.
    
    - `ConstantFolding`: remove the `Size(CreateMap)` case. When all children 
of the map are foldable, the generic `foldable` case still folds the 
expression, and it builds the map with `ArrayBasedMapBuilder` to get the real 
size. The `Size(CreateArray)` case is unchanged.
    - `CreateMap`: remove `NoThrow`. Building a map throws `NULL_MAP_KEY` on a 
NULL key, and `DUPLICATED_MAP_KEY` on a duplicate key under the default 
`EXCEPTION` policy. Because of `NoThrow`, `size(array(map(k, 1)))` was folded 
to `1` without building the map.
    - Add a migration guide entry.
    
    Folding the size only when the keys are distinct non-null literals was also 
considered. That check would have to repeat how `ArrayBasedMapBuilder` compares 
keys: `0.0` and `-0.0` are one key (and so are NaN values), strings with a 
non-binary collation are compared by their collation, and keys are often 
wrapped in a `Cast` after type coercion. It would only help a map with literal 
keys and non-literal values, such as `size(map('k1', a, 'k2', b))`. The plan 
that [SPARK-33544](https://is [...]
    
    ### Why are the changes needed?
    
    This is a correctness issue. The fold has been there since SPARK-33544 
([v3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1), 2021-02-22).
    
    ```sql
    SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
    
    -- Issue 1: It should return 1, but returns 2.
    SELECT size(map(1, 'a', 1, 'b'));
    
    -- Issue 2: It should return 1, but returns 2.
    CREATE TABLE t(a STRING, b STRING) USING parquet;
    INSERT INTO t VALUES ('x', 'x');
    SELECT cardinality(map(a, 1, b, 2)) FROM t;
    ```
    
    Under the default `EXCEPTION` policy, and for NULL keys under any policy, 
the fold also hides the error that building the map raises, so the result 
depends on whether `ConstantFolding` runs.
    
    ```sql
    SET spark.sql.mapKeyDedupPolicy=EXCEPTION;
    
    -- Issue 3: It should fail with DUPLICATED_MAP_KEY, but returns 2.
    SELECT size(map(1, 'a', 1, 'b'));
    
    -- Issue 4: It should fail with NULL_MAP_KEY, but returns 1.
    CREATE TABLE n(k STRING) USING parquet;
    INSERT INTO n VALUES (NULL);
    SELECT size(map(k, 1)) FROM n;
    
    -- Issue 5: It should fail with NULL_MAP_KEY, but returns 1.
    SELECT size(array(map(k, 1))) FROM n;
    ```
    
    With 
`spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding`,
 all of these queries already return the expected results.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `size` and `cardinality` of a map built by `map(...)` return the 
number of entries of the built map, as when `ConstantFolding` is excluded.
    
    - With `LAST_WIN`, a map with duplicate keys returns the deduplicated size, 
for example `1` instead of `2` for `size(map(1, 'a', 1, 'b'))`.
    - With `EXCEPTION`, a map with duplicate keys fails with 
`DUPLICATED_MAP_KEY` instead of returning the number of pairs.
    - A map with a NULL key fails with `NULL_MAP_KEY` instead of returning the 
number of pairs.
    
    `size(map(...))` with a non-foldable key or value is no longer folded 
during optimization, and is evaluated at runtime instead. The migration guide 
is updated.
    
    ### How was this patch tested?
    
    Pass the CIs with a newly added test case in `ConstantFoldingSuite`. It 
fails without this PR: the first part folds the size to `2` instead of `1`, and 
the second part folds `size(array(map(a, 1)))` to `1`.
    
    ```
    $ build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.*'
    $ build/sbt 'sql/testOnly *DataFrameFunctionsSuite'
    $ build/sbt 'sql/testOnly *SQLQueryTestSuite -- -z map.sql -z array.sql -z 
mapconcat.sql -z mapZipWith.sql -z distinct-map-aggregates.sql'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60067 from dongjoon-hyun/SPARK-60849.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit a475424a8c56adf84f23e8a6051a385dbb65c50b)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                             |  1 +
 .../sql/catalyst/expressions/complexTypeCreator.scala   |  2 +-
 .../spark/sql/catalyst/optimizer/expressions.scala      |  2 --
 .../sql/catalyst/optimizer/ConstantFoldingSuite.scala   | 17 +++++++++++++++++
 4 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 84c2714e2790..edb882b73634 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -26,6 +26,7 @@ license: |
 
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `DECIMAL` type from directory names, it converts each value to 
the precision and scale of that type, rounding half up as `CAST` does. 
Previously, the vectorized Parquet and ORC readers applied the declared scale 
to the digits in the directory name without an error, for example `d=1.5` read 
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results 
may need to be recomputed. A value that [...]
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out 
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of 
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped 
around without an error, for example `b=300` read as `TINYINT` returned `44`. 
Data derived from affected query results may need to be recomputed. To get 
`NULL` for such a value, set `spark. [...]
+- Since Spark 4.4, `size` and `cardinality` of a map built by `map(...)` 
return the number of entries of the built map. Previously, the optimizer could 
return the number of key-value pairs instead, for example `size(map(1, 'a', 1, 
'b'))` returned `2` instead of `1` when `spark.sql.mapKeyDedupPolicy` is 
`LAST_WIN`. Data derived from affected query results may need to be recomputed. 
Under the default `EXCEPTION` policy, such a query now fails with 
`DUPLICATED_MAP_KEY`, and a NULL key fails [...]
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory 
names, a value that cannot be cast to that type fails with 
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously, 
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it 
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such 
as `2020-01-01 25:00:00`, became the start of that da [...]
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
index 301281047721..acda793a8e77 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@@ -200,7 +200,7 @@ private [sql] object GenArrayData {
   since = "2.0.0",
   group = "map_funcs")
 case class CreateMap(children: Seq[Expression], useStringTypeWhenEmpty: 
Boolean)
-  extends Expression with NoThrow {
+  extends Expression {
 
   def this(children: Seq[Expression]) = {
     this(children, 
SQLConf.get.getConf(SQLConf.LEGACY_CREATE_EMPTY_COLLECTION_USING_STRING_TYPE))
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 9e8c5fba4135..78ccafd6f027 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -92,8 +92,6 @@ object ConstantFolding extends Rule[LogicalPlan] {
 
     case Size(c: CreateArray, _) if c.children.forall(hasNoSideEffect) =>
       Literal(c.children.length)
-    case Size(c: CreateMap, _) if c.children.forall(hasNoSideEffect) =>
-      Literal(c.children.length / 2)
 
     case e if e.containsTag(FAILED_TO_EVALUATE) => e
 
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
index e160accec9ac..997996d28965 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
@@ -342,6 +342,23 @@ class ConstantFoldingSuite extends PlanTest {
     }
   }
 
+  test("SPARK-60849: Constant folding test CreateMap") {
+    // Building a map deduplicates its keys, so its size can be less than the 
number of pairs.
+    withSQLConf(SQLConf.MAP_KEY_DEDUP_POLICY.key -> 
SQLConf.MapKeyDedupPolicy.LAST_WIN.toString) {
+      val originalQuery = testRelation
+        .select(Size(CreateMap(Seq(Literal(1), Literal(1), Literal(1), 
Literal(2)))).as("s"))
+      comparePlans(
+        Optimize.execute(originalQuery.analyze),
+        testRelation.select(Literal(1).as("s")).analyze)
+    }
+
+    // Building a map can throw, for example on a NULL key.
+    val originalQuery = testRelation
+      .select(Size(CreateArray(Seq(CreateMap(Seq($"a", Literal(1)))))).as("s"))
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), originalQuery)
+  }
+
   test("SPARK-37907: InvokeLike support ConstantFolding") {
     val originalQuery =
       testRelation


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to