srielau opened a new pull request, #59325:
URL: https://github.com/apache/spark/pull/59325
### What changes were proposed in this pull request?
When `spark.sql.charVarchar.standardSemantics.enabled` is true, `from_json`
and the JSON datasource length-check JSON object names used as `MAP<CHAR(n),
_>` / `MAP<VARCHAR(n), _>` keys without rewriting them.
- `CHAR(n)` keys must already be exactly `n` characters (no pad).
- `VARCHAR(n)` keys must already be at most `n` characters (no trim).
- Exact repeated member names last-win. `spark.sql.mapKeyDedupPolicy` is not
applied.
- Length failures follow the JSON parse mode (`PERMISSIVE` / `FAILFAST`).
Mismatched keys raise a new `UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY`
condition (SQLSTATE `0A000`) instead of `EXCEED_LIMIT_LENGTH`, because pad/trim
of map keys is not supported. JSON map *values* still use
`EXCEED_LIMIT_LENGTH`. CSV, XML, and TRANSFORM are unchanged.
### Why are the changes needed?
SPARK-59274 already applies CHAR pad and VARCHAR overflow to JSON *values*.
Map keys were still taken as the raw object names. Applying write-side pad/trim
to keys would rewrite names and could invent collisions, so this change only
validates length.
### Does this PR introduce _any_ user-facing change?
Yes, under `spark.sql.charVarchar.standardSemantics.enabled=true`:
```sql
SELECT from_json('{"a": 1}', 'MAP<CHAR(3), INT>', map('mode', 'FAILFAST'));
-- [UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY] JSON map keys of CHAR or VARCHAR
cannot be padded or trimmed.
-- The key 'a' is not valid for type "CHAR(3)". SQLSTATE: 0A000
```
Exact-width CHAR keys and in-limit VARCHAR keys are kept as-is, including
trailing spaces. With the flag off, keys remain the raw object names.
### How was this patch tested?
Added SPARK-60108 coverage in `BasicCharVarcharTestSuite` (too-short CHAR,
exact-width CHAR with spaces, CHAR/VARCHAR overflow, distinct VARCHAR `"ab"` vs
`"ab "`, last-wins duplicates, PERMISSIVE vs FAILFAST, nested maps, `from_json`
and the JSON datasource). Also re-ran the SPARK-59274 JSON map value overflow
tests and `SparkThrowableSuite` error-condition formatting.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
