Serge Rielau created SPARK-60229:
------------------------------------
Summary: [SQL] from_json CHAR/VARCHAR map-key check can be skipped
by OptimizeCsvJsonExprs and can mis-assign nested fields when partial results
are off
Key: SPARK-60229
URL: https://issues.apache.org/jira/browse/SPARK-60229
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
Follow-up to SPARK-60108
(https://issues.apache.org/jira/browse/SPARK-60108,
https://github.com/apache/spark/pull/59325).
SPARK-60108 length-checks JSON object names used as CHAR/VARCHAR map keys
(no pad/trim). Two existing paths can still bypass that check or corrupt
an enclosing struct.
1. OptimizeCsvJsonExprs still rewrites
from_json(to_json(m), same_map_type) => m
whenever options are empty and the types match. That was a safe
round-trip before SPARK-60108. It is not safe once from_json validates
CHAR/VARCHAR keys.
Repro (flag on at the from_json; m already typed MAP<CHAR(3), INT>
with an unpadded key, e.g. from a view created with the flag off):
SET spark.sql.charVarchar.standardSemantics.enabled = true;
SELECT from_json(to_json(m), 'MAP<CHAR(3), INT>') FROM t;
Expected: NULL in PERMISSIVE (key "a" is not CHAR(3)); FAILFAST raises
UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY.
Actual: optimizer returns m unchanged, so the invalid key survives.
Fix: do not apply the rewrite when standard CHAR/VARCHAR semantics are
captured on JsonToStructs and the schema contains a CHAR/VARCHAR map
key (any nesting).
2. Nested convertObject can steal an inner field name when
spark.sql.json.enablePartialResults is false (internal, default true).
SPARK-60108 already consumes a rejected map key before returning, and
tests a sibling tail next to a top-level map:
from_json('{"m":{"a":1},"tail":1}', 'm MAP<CHAR(3),INT>, tail INT')
That does not cover a rejected map nested inside a struct that also has
a same-named sibling, while the root struct has the same field:
SET spark.sql.json.enablePartialResults = false;
SET spark.sql.charVarchar.standardSemantics.enabled = true;
SELECT from_json(
'{"s":{"m":{"a":1},"tail":5},"tail":1}',
's STRUCT<m: MAP<CHAR(3), INT>, tail: INT>, tail INT');
convertMap throws while the parser is still inside s. The root
convertObject catch (isRoot || enablePartialResults) then skipChildren()
from a nested token, so inner "tail":5 can be applied to the outer tail.
Expected: s.m NULL (or the struct nulled), outer tail = 1.
Actual: outer tail may be 5.
Fix: a nested map/struct converter that fails non-partially must still
consume its own object through END_OBJECT before the exception
propagates, matching the SPARK-60108 FIELD_NAME comment on convertMap.
Tests
- from_json(to_json(MAP<CHAR(n)/VARCHAR(n), _>)) must not fold away under
CHAR_VARCHAR_STANDARD_SEMANTICS; PERMISSIVE and FAILFAST.
- Nested struct+map+duplicate field name with
JSON_ENABLE_PARTIAL_RESULTS = false and true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
