[
https://issues.apache.org/jira/browse/SPARK-60229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-60229:
-----------------------------------
Affects Version/s: 4.4.0
> [SQL] from_json CHAR/VARCHAR map-key check can be skipped by
> OptimizeCsvJsonExprs and can mis-assign nested fields when partial results
> are off
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60229
> URL: https://issues.apache.org/jira/browse/SPARK-60229
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Serge Rielau
> Assignee: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> Follow-up to SPARK-60108
> (https://issues.apache.org/jira/browse/SPARK-60108,
> https://github.com/apache/spark/pull/59325).
> SPARK-60108 length-checks JSON object names used as CHAR/VARCHAR map keys
> (no pad/trim). Two existing paths can still bypass that check or corrupt
> an enclosing struct.
> 1. OptimizeCsvJsonExprs still rewrites
> from_json(to_json(m), same_map_type) => m
> whenever options are empty and the types match. That was a safe
> round-trip before SPARK-60108. It is not safe once from_json validates
> CHAR/VARCHAR keys.
> Repro (flag on at the from_json; m already typed MAP<CHAR(3), INT>
> with an unpadded key, e.g. from a view created with the flag off):
> SET spark.sql.charVarchar.standardSemantics.enabled = true;
> SELECT from_json(to_json(m), 'MAP<CHAR(3), INT>') FROM t;
> Expected: NULL in PERMISSIVE (key "a" is not CHAR(3)); FAILFAST raises
> UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY.
> Actual: optimizer returns m unchanged, so the invalid key survives.
> Fix: do not apply the rewrite when standard CHAR/VARCHAR semantics are
> captured on JsonToStructs and the schema contains a CHAR/VARCHAR map
> key (any nesting).
> 2. Nested convertObject can steal an inner field name when
> spark.sql.json.enablePartialResults is false (internal, default true).
> SPARK-60108 already consumes a rejected map key before returning, and
> tests a sibling tail next to a top-level map:
> from_json('{"m":{"a":1},"tail":1}', 'm MAP<CHAR(3),INT>, tail INT')
> That does not cover a rejected map nested inside a struct that also has
> a same-named sibling, while the root struct has the same field:
> SET spark.sql.json.enablePartialResults = false;
> SET spark.sql.charVarchar.standardSemantics.enabled = true;
> SELECT from_json(
> '{"s":{"m":{"a":1},"tail":5},"tail":1}',
> 's STRUCT<m: MAP<CHAR(3), INT>, tail: INT>, tail INT');
> convertMap throws while the parser is still inside s. The root
> convertObject catch (isRoot || enablePartialResults) then skipChildren()
> from a nested token, so inner "tail":5 can be applied to the outer tail.
> Expected: s.m NULL (or the struct nulled), outer tail = 1.
> Actual: outer tail may be 5.
> Fix: a nested map/struct converter that fails non-partially must still
> consume its own object through END_OBJECT before the exception
> propagates, matching the SPARK-60108 FIELD_NAME comment on convertMap.
> Tests
> - from_json(to_json(MAP<CHAR(n)/VARCHAR(n), _>)) must not fold away under
> CHAR_VARCHAR_STANDARD_SEMANTICS; PERMISSIVE and FAILFAST.
> - Nested struct+map+duplicate field name with
> JSON_ENABLE_PARTIAL_RESULTS = false and true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
