Serge Rielau created SPARK-59722:
------------------------------------
Summary: Enforce CHAR/VARCHAR semantics for JSON map keys in
schema-driven parsing
Key: SPARK-59722
URL: https://issues.apache.org/jira/browse/SPARK-59722
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
{{from_json}} / JSON datasource maps with a first-class {{MAP<CHAR(n), ...>}}
or {{MAP<VARCHAR(n), ...>}} schema still treat keys as unbounded STRING.
Distinct serialized names can therefore collide after CHAR padding or VARCHAR
trailing-space trimming ({{{}"a"{}}} vs {{"a "}} as {{{}CHAR(2){}}}), producing
duplicate physical keys or silent last-wins instead of
{{{}spark.sql.mapKeyDedupPolicy{}}}.
SPARK-59274 covers scalar CHAR/VARCHAR assignment in JSON/CSV/XML,
{{{}schema_of_*{}}}, and XML map keys. JSON map-key normalization is the
remaining CHAR/VARCHAR surface.
In scope
* Apply assignment semantics to JSON object names when the declared map key
type is CHAR/VARCHAR (pad CHAR; overflow -> {{{}EXCEED_LIMIT_LENGTH{}}}).
* Honor {{mapKeyDedupPolicy}} for collisions created by that normalization;
keep exact repeated names as historical last-wins.
* Keep ordinary {{MAP<STRING, ...>}} JSON construction unchanged (including
duplicate-entry cardinality).
* Drain a failed nested JSON map value to its owning boundary before recovery
continues (required so CHAR/VARCHAR key handling does not leak inner fields as
outer keys).
Out of scope
* Ordinary STRING maps, including non-binary collated keys ({{{}UTF8_LCASE{}}}
{{a}} vs {{{}A{}}}). That is pre-existing parser/collation behavior, not
CHAR/VARCHAR length semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
