Qing Fu created SPARK-60070:
-------------------------------
Summary: File-format readers can produce MapData with null keys
Key: SPARK-60070
URL: https://issues.apache.org/jira/browse/SPARK-60070
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Qing Fu
{{ArrayBasedMapBuilder}} rejects null map keys, but several readers construct
{{ArrayBasedMapData}} directly without a runtime check, so a null key can reach
query execution:
* {{ParquetRowConverter}} (map converter {{end()}}): the in-tree comment says
the map "may contains null or duplicated map keys ... the behavior is
undefined" and points to SPARK-26174, which was resolved as Incomplete in 2021
without a fix.
* {{OrcDeserializer}} and {{AvroDeserializer}} build the key array without a
null check (each asserts in a comment that the format cannot produce one).
* {{HiveInspectors}} applies the {{MapObjectInspector}} key unwrapper with no
null check.
SPARK-59598 makes {{GetMapValue}} / {{ElementAt}} skip null keys, but other
consumers still see the malformed map: {{to_json}}, the CSV and XML generators,
and {{MapZipWith}} (which treats two null keys as equal and mergeable).
Proposal for discussion: reject null keys at the reader boundary with a clear
error, behind a legacy config that restores the current behavior. Open
questions: the default (fail vs. drop the entry), and whether duplicate keys
should be handled in the same change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
