srielau opened a new pull request, #58033: URL: https://github.com/apache/spark/pull/58033
### What changes were proposed in this pull request? This is the foundation for SQL standard–aligned `CHAR(n)` / `VARCHAR(n)` under parent [SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794) (subtasks SPARK-58796–SPARK-58801). Adds `spark.sql.charVarchar.standardSemantics.enabled` (default `false`). When enabled (or together with `preserveCharVarcharTypeInfo` for first-class typing): - **SPARK-58796**: CHAR/VARCHAR may appear as first-class types in schemas/plans (`charVarcharFirstClassTypes`) - **SPARK-58797**: CAST / try_cast keep CHAR/VARCHAR targets and apply character-length pad/trim/`EXCEED_LIMIT_LENGTH` - **SPARK-58798**: LCT lattice `CHAR → VARCHAR → STRING` with `max(n, m)` for COALESCE/CASE/UNION/etc. - **SPARK-58799**: Transforming string expressions (`upper`, `substr`, `concat`/`||`, …) return plain `STRING` (R1) - **SPARK-58800**: Write-side checks return CharType/VarcharType when first-class - **SPARK-58801**: Read-side CHAR padding forced under standardSemantics; oversize after trailing-blank trim errors `preserveCharVarcharTypeInfo` remains the Spark 4.0 experimental leaky path; only `standardSemantics` applies R1. ### Why are the changes needed? Today CHAR/VARCHAR are largely annotated STRING: CAST stringifies targets, LCT always widens to STRING when preserve is off, and transforming ops can leak length types when preserve is on. Standard semantics need a gated, consistent path. ### Does this PR introduce _any_ user-facing change? Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is set to `true` (default remains `false`): - CAST AS CHAR/VARCHAR preserves typed results and enforces length - COALESCE/CASE/UNION may return CHAR/VARCHAR - string transforms return STRING - scans pad CHAR and reject oversize values ### How was this patch tested? - New cases in `CharVarcharTestSuite` / `BasicCharVarcharTestSuite` / `FileSourceCharVarcharTestSuite` covering CAST, LCT, R1, createDataFrame, and scan pad/oversize - Local SBT run was blocked by Maven Central network access in this environment; please verify with: ``` ./build/sbt "sql/testOnly *BasicCharVarcharTestSuite *FileSourceCharVarcharTestSuite" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
