Yan Yan created SPARK-61608:
-------------------------------
Summary: Fix arrays_zip field naming for map lookups
Key: SPARK-61608
URL: https://issues.apache.org/jira/browse/SPARK-61608
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yan Yan
arrays_zip fails for valid array-valued map lookups when the key is computed,
NULL, or a non-string literal. Its field-name inference tries to turn the key
expression into a literal output field name, causing expression or
result-schema construction to fail.
Self-contained example:
{code:sql}
SELECT to_json(arrays_zip(map(1, array(10))[1]));
{code}
Expected result: [{"0":10}]. The map lookup selects array(10); arrays_zip uses
the first argument's position, 0, as the output field name. Before the fix, the
query fails while constructing the expression/schema.
The same problem affects row-dependent keys, for example:
{code:sql}
SELECT arrays_zip(m[k])
FROM VALUES (map('a', array(10)), 'a') AS t(m, k);
{code}
The proposed fix uses a map key as a field name only when the resolved key is a
non-null string literal. Other key expressions use the existing zero-based
argument position. Map lookup evaluation remains unchanged. Existing direct
string-literal names and explicit aliases are preserved.
Reproduced against current Apache Spark master before changing production code;
focused Catalyst regressions showed eight behavioral failures. After the fix,
the full CollectionExpressionsSuite passes 77 tests and the SQL arrays_zip
tests pass 9 tests, including generated/interpreted evaluation and a
stored-view collation case across analyzer modes. A Spark 4.4 migration-guide
note describes the behavior.
Pull request: https://github.com/apache/spark/pull/60356
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]