Yan Yan created SPARK-61608:
-------------------------------

             Summary: Fix arrays_zip field naming for map lookups
                 Key: SPARK-61608
                 URL: https://issues.apache.org/jira/browse/SPARK-61608
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yan Yan


arrays_zip fails for valid array-valued map lookups when the key is computed, 
NULL, or a non-string literal. Its field-name inference tries to turn the key 
expression into a literal output field name, causing expression or 
result-schema construction to fail.

Self-contained example:
{code:sql}
SELECT to_json(arrays_zip(map(1, array(10))[1]));
{code}
Expected result: [{"0":10}]. The map lookup selects array(10); arrays_zip uses 
the first argument's position, 0, as the output field name. Before the fix, the 
query fails while constructing the expression/schema.

The same problem affects row-dependent keys, for example:
{code:sql}
SELECT arrays_zip(m[k])
FROM VALUES (map('a', array(10)), 'a') AS t(m, k);
{code}

The proposed fix uses a map key as a field name only when the resolved key is a 
non-null string literal. Other key expressions use the existing zero-based 
argument position. Map lookup evaluation remains unchanged. Existing direct 
string-literal names and explicit aliases are preserved.

Reproduced against current Apache Spark master before changing production code; 
focused Catalyst regressions showed eight behavioral failures. After the fix, 
the full CollectionExpressionsSuite passes 77 tests and the SQL arrays_zip 
tests pass 9 tests, including generated/interpreted evaluation and a 
stored-view collation case across analyzer modes. A Spark 4.4 migration-guide 
note describes the behavior.

Pull request: https://github.com/apache/spark/pull/60356




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to