srielau opened a new pull request, #58059: URL: https://github.com/apache/spark/pull/58059
### What changes were proposed in this pull request? **Depends on #58033** ([SPARK-58794](https://issues.apache.org/jira/browse/SPARK-58794) foundation). Please review/merge that first; rebase this onto `master` afterward. Delta vs foundation: https://github.com/srielau/spark/compare/SPARK-58794...SPARK-58803 [SPARK-58803](https://issues.apache.org/jira/browse/SPARK-58803): prove Dataset / encoder / UDF CHAR/VARCHAR surfaces under `spark.sql.charVarchar.standardSemantics.enabled`. The foundation already gates these APIs via `charVarcharFirstClassTypes` (`failIfHasCharVarchar`, `RowEncoder` → `CharEncoder`/`VarcharEncoder`, `CatalystTypeConverters` Char/Varchar converters) and applies write-side pad/trim/`EXCEED_LIMIT_LENGTH`. This PR adds the end-to-end coverage that belongs with that API contract: - `createDataFrame` / `Encoders.CHAR` / `Encoders.VARCHAR`: pad + oversize - `spark.udf.register` and Java `udf(..., returnType)`: typed results + length enforcement - `Dataset.to`: CHAR/VARCHAR target schema + narrowing assignment - `DataFrameReader` / `DataStreamReader` user schemas keep CHAR/VARCHAR - `RowEncoderSuite` / `UDFSuite`: standardSemantics path (not only `preserveCharVarcharTypeInfo`) ### Why are the changes needed? Without dedicated tests, Dataset/UDF/reader CHAR/VARCHAR under standardSemantics is only incidentally covered. SPARK-58803 is the API-surface closure for Phase 2. ### Does this PR introduce _any_ user-facing change? No new behavior beyond what #58033 already enables when the flag is on; this PR adds tests. ### How was this patch tested? ``` ./build/sbt "sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z SPARK-58803" ./build/sbt "sql/testOnly org.apache.spark.sql.UDFSuite -- -z \"char/varchar as UDF return type\"" ./build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.encoders.RowEncoderSuite -- -z SPARK-58803" ./build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.encoders.RowEncoderSuite -- -z char/varchar" ``` All of the above succeeded locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
