srielau opened a new pull request, #58541:
URL: https://github.com/apache/spark/pull/58541
### What changes were proposed in this pull request?
When `spark.sql.charVarchar.standardSemantics.enabled` is true, `CharType`
and `VarcharType` are first-class `StringType` subtypes. Several
execution-boundary matchers still used exact `StringType` (or an explicit
CHAR/VARCHAR reject), so those paths failed or skipped constrained string
columns.
This patch treats CHAR/VARCHAR as the string family at:
- JDBC getters/setters and JDBC array element conversion
- JDBC schema inference (`CHAR`/`VARCHAR` keep first-class types when
standard semantics is on, even if `charVarcharAsString` is also set)
- File partition value decoding
- `RowToColumnConverter`
- `DataFrame.na.fill` for string replacement values
- `ANALYZE TABLE ... FOR COLUMNS` (string-family stats)
Read-side CHAR padding and VARCHAR overflow still come from existing CAST /
`ApplyCharTypePadding` paths rather than being reimplemented in each converter.
### Why are the changes needed?
With first-class CHAR/VARCHAR, JDBC scans/writes, file-only partition
discovery, columnar conversion, `na.fill("...")`, and column stats currently
throw or silently ignore those columns. That blocks enabling standard semantics.
JIRA: https://issues.apache.org/jira/browse/SPARK-59273 (subtask of
SPARK-58794)
### Does this PR introduce _any_ user-facing change?
Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true (still
default false):
- JDBC read/write of CHAR/VARCHAR (including arrays) no longer fails with an
unsupported JDBC type.
- File partition columns declared as CHAR/VARCHAR can be decoded; CHAR is
padded on scan and oversize VARCHAR fails with `EXCEED_LIMIT_LENGTH`.
- Columnar row-to-column conversion accepts CHAR/VARCHAR.
- `df.na.fill("x")` fills null CHAR/VARCHAR columns (CHAR values are padded
by CAST).
- `ANALYZE TABLE ... FOR COLUMNS` collects string-family stats on
CHAR/VARCHAR instead of rejecting them.
### How was this patch tested?
Added/extended unit tests:
- `JDBCSuite`: read CHAR/VARCHAR and arrays; write CHAR/VARCHAR; standard
semantics wins over `charVarcharAsString` in schema inference
- `ParquetV1PartitionDiscoverySuite` / `ParquetV2PartitionDiscoverySuite`:
CHAR/VARCHAR partition values and oversize VARCHAR
- `RowToColumnConverterSuite`: CHAR/VARCHAR and nested CHAR arrays
- `DataFrameNaFunctionsSuite`: `na.fill` on CHAR/VARCHAR
- `StatisticsCollectionSuite`: `ANALYZE TABLE ... FOR COLUMNS` on
CHAR/VARCHAR
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
