srielau commented on code in PR #59302: URL: https://github.com/apache/spark/pull/59302#discussion_r4240445764
########## docs/sql-migration-guide.md: ########## @@ -29,6 +29,7 @@ license: | ## Upgrading from Spark SQL 4.3 to 4.4 - Since Spark 4.4, when `spark.sql.preserveCharVarcharTypeInfo` is true and `spark.sql.charVarchar.standardSemantics.enabled` is false, ORC reads that apply a CHAR/VARCHAR schema over STRING storage return the stored values without ORC truncation, matching Parquet. Previously the ORC reader requested `char(n)`/`varchar(n)` and truncated STRING-stored values to `n`. Read-side length checks (`EXCEED_LIMIT_LENGTH`) apply only when `spark.sql.charVarchar.standardSemantics.enabled` is true. +- Since Spark 4.4, Hive `TRANSFORM` / `MAP` / `REDUCE` with CHAR/VARCHAR `AS` types keeps script I/O as unbounded STRING and applies Spark assignment (CHAR padding and `EXCEED_LIMIT_LENGTH` on overflow) in a `Project` whenever first-class CHAR/VARCHAR is on. That includes preserve-only (`spark.sql.preserveCharVarcharTypeInfo=true` without standard semantics). Previously preserve-only left no-SerDe TRANSFORM output as STRING, and Hive SerDe deserialized `char(n)`/`varchar(n)` with Hive truncation. With both flags off, `AS (c CHAR(n))` remains STRING with no pad and no length check. Review Comment: Done. The migration bullet now names both `spark.sql.charVarchar.standardSemantics.enabled` and `spark.sql.preserveCharVarcharTypeInfo`, and the prior-behavior text matches branch-4.3: scalar no-SerDe CHAR/VARCHAR failed with the unsupported-type error, nested no-SerDe kept the declared type with no pad or length check, and Hive SerDe still used Hive truncation. Rebased onto current `master` (kept the new XML CHAR/VARCHAR map-key bullet next to it). PR description updated the same way, and SPARK-59683 is described as unreleased master / branch-4.x rather than 4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
