srielau opened a new pull request, #59167: URL: https://github.com/apache/spark/pull/59167
### What changes were proposed in this pull request? Follow-up to SPARK-59274. Schema-driven JSON (`from_json` and the JSON datasource) still treated `MAP<CHAR, ...>` / `MAP<VARCHAR, ...>` object names as unbounded STRING. Distinct serialized names could therefore collide after CHAR padding or VARCHAR trailing-space trimming (`"a"` vs `"a "` as `CHAR(2)`), producing duplicate physical keys or silent last-wins instead of honoring `spark.sql.mapKeyDedupPolicy`. This patch applies assignment semantics to JSON object names when the declared map key type is CHAR/VARCHAR: - Pad CHAR keys; oversize VARCHAR/CHAR keys raise `EXCEED_LIMIT_LENGTH`. - Exact repeated names stay historical last-wins. - Collisions created by that normalization honor `spark.sql.mapKeyDedupPolicy`. - Ordinary `MAP<STRING, ...>` construction is unchanged, including duplicate-entry cardinality. - Failed nested JSON map values drain to their owning boundary so inner fields cannot leak as outer keys. - `DUPLICATED_MAP_KEY` is not swallowed as a corrupt JSON record. JIRA: https://issues.apache.org/jira/browse/SPARK-59722 ### Why are the changes needed? SPARK-59274 covered scalar CHAR/VARCHAR assignment in JSON/CSV/XML, `schema_of_*`, and XML map keys. JSON map-key normalization was the remaining CHAR/VARCHAR surface. Without this, enabling `spark.sql.charVarchar.standardSemantics.enabled` still drops length rules on JSON map keys. ### Does this PR introduce _any_ user-facing change? Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true (still default false): - `from_json('{"ab": 1}', 'MAP<CHAR(4), INT>')` pads the key to `"ab "`. - `from_json('{"a":1,"a ":2}', 'MAP<CHAR(2), INT>')` raises `DUPLICATED_MAP_KEY` under EXCEPTION and keeps `"a " -> 2` under LAST_WIN. - Exact `{"a":1,"a":2}` as `MAP<CHAR(2), INT>` remains last-wins regardless of policy. - `MAP<STRING, INT>` `{"a":1,"a":2}` still has size 2. Default (flag off) is unchanged: `from_json(..., 'MAP<CHAR(5), INT>')` still fails with `UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING`. ### How was this patch tested? Added `BasicCharVarcharTestSuite` coverage for SPARK-59722: CHAR padding, VARCHAR overflow, normalized-key collisions vs exact-name last-wins, nested maps, sibling fields after a failed map, pretty-printed drain, JSON datasource reads, and collated CHAR/VARCHAR keys. Ran: ``` sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z SPARK-59722 sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "honor the dedup policy" sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "JSON map value overflow" sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "ordinary STRING map duplicate" sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "from_json/csv/xml honor CHAR" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
