srielau opened a new pull request, #58549:
URL: https://github.com/apache/spark/pull/58549
### What changes were proposed in this pull request?
Complete first-class CHAR/VARCHAR support at PySpark and Arrow boundaries
when
`spark.sql.charVarchar.standardSemantics.enabled` is true:
- Map Python `CharType` and `VarcharType` to Arrow UTF8, recursively through
complex types.
- Apply CHAR padding and VARCHAR length checks to pickled, Arrow-optimized,
and pandas UDF
results before they re-enter Catalyst.
- Compare Arrow UDF output using its physical STRING representation while
retaining logical
CHAR/VARCHAR in the Spark schema.
- Apply the same recursive assignment checks to local, pandas, and PyArrow
explicit-schema
DataFrame creation.
- Allow `DataFrame.toArrow()` to export CHAR/VARCHAR values as Arrow strings.
- Handle CHAR/VARCHAR output in the Arrow columnar-input execution path.
JIRA: https://issues.apache.org/jira/browse/SPARK-59275
### Why are the changes needed?
CHAR/VARCHAR are first-class types under standard semantics, but Python and
Arrow boundaries
still treated them inconsistently. Arrow UDFs rejected them as unsupported,
pickled UDF and
explicit-schema creation paths did not enforce their length rules, and Arrow
output validation
compared logical CHAR/VARCHAR against physical STRING.
These gaps allowed unpadded CHAR and over-length VARCHAR values or caused
supported queries to
fail. The checks must also recurse through structs, arrays, and maps.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.charVarchar.standardSemantics.enabled=true`, Python
UDFs, Arrow-optimized
UDFs, pandas UDFs, and explicit-schema DataFrame creation now accept
CHAR/VARCHAR and enforce
their assignment semantics. `toArrow()` exports these values as Arrow
strings. The default
flag-off behavior is unchanged.
### How was this patch tested?
Added PySpark coverage for:
- Non-Arrow, Arrow-optimized, and pandas UDF CHAR/VARCHAR results.
- CHAR padding, VARCHAR overflow, and nested struct/array/map results.
- Arrow columnar input.
- Local, pandas, and PyArrow explicit-schema DataFrame creation.
- `DataFrame.toArrow()`.
Ran:
```
/usr/bin/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
-Dsbt.override.build.repos=true 'sql/Test/compile'
/usr/bin/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
-Dsbt.override.build.repos=true \
'sql/testOnly org.apache.spark.sql.execution.python.EvaluatePythonSuite --
-z SPARK-59275'
python3 -m py_compile \
python/pyspark/sql/pandas/types.py \
python/pyspark/sql/tests/test_udf.py \
python/pyspark/sql/tests/test_creation.py \
python/pyspark/sql/tests/arrow/test_arrow.py \
python/pyspark/sql/tests/arrow/test_arrow_python_udf.py
```
The PySpark integration tests were not run locally because PyArrow is not
installed in this
environment.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Auto (GPT-5.6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
