HyukjinKwon commented on code in PR #59302: URL: https://github.com/apache/spark/pull/59302#discussion_r4238585252
########## docs/sql-migration-guide.md: ########## @@ -29,6 +29,7 @@ license: | ## Upgrading from Spark SQL 4.3 to 4.4 - Since Spark 4.4, when `spark.sql.preserveCharVarcharTypeInfo` is true and `spark.sql.charVarchar.standardSemantics.enabled` is false, ORC reads that apply a CHAR/VARCHAR schema over STRING storage return the stored values without ORC truncation, matching Parquet. Previously the ORC reader requested `char(n)`/`varchar(n)` and truncated STRING-stored values to `n`. Read-side length checks (`EXCEED_LIMIT_LENGTH`) apply only when `spark.sql.charVarchar.standardSemantics.enabled` is true. +- Since Spark 4.4, Hive `TRANSFORM` / `MAP` / `REDUCE` with CHAR/VARCHAR `AS` types keeps script I/O as unbounded STRING and applies Spark assignment (CHAR padding and `EXCEED_LIMIT_LENGTH` on overflow) in a `Project` whenever first-class CHAR/VARCHAR is on. That includes preserve-only (`spark.sql.preserveCharVarcharTypeInfo=true` without standard semantics). Previously preserve-only left no-SerDe TRANSFORM output as STRING, and Hive SerDe deserialized `char(n)`/`varchar(n)` with Hive truncation. With both flags off, `AS (c CHAR(n))` remains STRING with no pad and no length check. Review Comment: **Nit (P3):** This sentence does not match branch-4.3: "Previously preserve-only left no-SerDe TRANSFORM output as STRING". With only `spark.sql.preserveCharVarcharTypeInfo=true`, the `AS` schema kept `CharType`/`VarcharType`, so without a SerDe: - scalar `AS (c CHAR(4))` failed in the script reader with the unsupported-output-type error; - nested types such as `ARRAY<CHAR(4)>` came back CHAR-typed, with no padding or length check. Neither case returned STRING. A preserve-only user reading this note will not recognize the change they are getting: their scalar queries used to fail, and their nested results were already CHAR-typed. The Hive SerDe half of the sentence is accurate. One possible rewrite, which also names both configs (the missing `spark.sql.charVarchar.standardSemantics.enabled` is covered in the review summary): ```suggestion - Since Spark 4.4, Hive `TRANSFORM` / `MAP` / `REDUCE` with CHAR/VARCHAR `AS` types keeps script I/O as unbounded STRING and applies Spark assignment (CHAR padding and `EXCEED_LIMIT_LENGTH` on overflow) in a `Project` when `spark.sql.charVarchar.standardSemantics.enabled` or `spark.sql.preserveCharVarcharTypeInfo` is true. Previously, with only `spark.sql.preserveCharVarcharTypeInfo` set, scalar CHAR/VARCHAR output without a SerDe failed with an unsupported-type error, nested CHAR/VARCHAR output without a SerDe kept the declared type with no padding or length check, and Hive SerDe deserialized `char(n)`/`varchar(n)` with Hive truncation. When both configs are false, `AS (c CHAR(n))` remains STRING with no pad and no length check. ``` See **Shared repair plan 1** in the review body. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
