srielau opened a new pull request, #59468:
URL: https://github.com/apache/spark/pull/59468
### What changes were proposed in this pull request?
Apply CHAR/VARCHAR assignment in `from_avro` / `to_avro` the same way
`from_json` / `from_csv` / `from_xml` already do, via
`CharVarcharUtils.applyTextParseSemantics`.
`AvroDeserializer` and `AvroSerializer` (STRING, ENUM, and CHAR map keys)
now pad CHAR to the declared length and reject values that still exceed the
limit after trailing-space trim (`EXCEED_LIMIT_LENGTH`).
### Why are the changes needed?
Avro already stamps `CharType` / `VarcharType` on the schema
(`spark.sql.catalyst.type`), but decode/encode copied UTF-8 as-is. A Spark
round-trip looked fine because writes were already checked. Externally produced
Avro could advertise CHAR/VARCHAR without padding or length checks.
Parent: SPARK-58794.
### Does this PR introduce _any_ user-facing change?
Yes, under `spark.sql.charVarchar.standardSemantics.enabled` (or whenever
the Catalyst type is CHAR/VARCHAR):
- `from_avro` of a short CHAR value is padded, matching CAST/store
assignment.
- `from_avro` / `to_avro` of an oversize CHAR/VARCHAR value fail with
`EXCEED_LIMIT_LENGTH` (default FAILFAST). Unbounded STRING is unchanged.
This is relative to current unreleased master CHAR/VARCHAR work, not
previously released Spark versions.
### How was this patch tested?
- `BasicCharVarcharTestSuite` (`SPARK-58814: major formats...`), including
in-flight Avro ser/de pad and oversize cases
- `AvroFunctionsSuite` (`SPARK-60242: from_avro/to_avro CHAR/VARCHAR length
enforcement`)
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt
-Dsbt.override.build.repos=true \
'sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z
"SPARK-58814: major formats"' \
'avro/testOnly org.apache.spark.sql.avro.AvroFunctionsSuite -- -z
"SPARK-60242"'
```
Scala lint (`./dev/lint-scala`) passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
