[ 
https://issues.apache.org/jira/browse/SPARK-61588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61588:
-----------------------------------
    Labels: pull-request-available  (was: )

> Preserve persisted string defaults in legacy escaping mode
> ----------------------------------------------------------
>
>                 Key: SPARK-61588
>                 URL: https://issues.apache.org/jira/browse/SPARK-61588
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Yan Yan
>            Priority: Minor
>              Labels: pull-request-available
>
> With spark.sql.parser.escapedStringLiterals=true, Spark can reconstruct a 
> persisted column default with extra backslashes. Rows written before ALTER 
> TABLE ADD COLUMN can therefore return a different value from rows inserted 
> later using DEFAULT.
> h3. Reproduction
> Run these statements in a fresh database. chr(39) constructs the apostrophe 
> without depending on SQL string escaping in the reproduction itself.
> {code:sql}
> SET spark.sql.parser.escapedStringLiterals=true;
> CREATE TABLE default_escape_repro (id INT) USING parquet;
> INSERT INTO default_escape_repro VALUES (1);
> ALTER TABLE default_escape_repro ADD COLUMNS (
>   s STRING DEFAULT concat('Dianne', chr(39), 's horse')
> );
> INSERT INTO default_escape_repro VALUES (2, DEFAULT);
> SELECT id, s, hex(s) FROM default_escape_repro ORDER BY id;
> {code}
> Expected: both rows contain {{Dianne's horse}}, with hex 
> {{4469616E6E65277320686F727365}}.
> Actual before the fix: the old row can contain {{Dianne\'s horse}}, with hex 
> {{4469616E6E655C277320686F727365}}, while the new row contains the expected 
> value.
> Strings containing literal backslashes are also affected. The problem extends 
> to strings nested in ARRAY, MAP and STRUCT defaults and to persisted default 
> values converted for V2 catalogs, including CREATE TABLE LIKE.
> h3. Cause and fix
> Spark folds a default and stores its value as SQL in EXISTS_DEFAULT metadata. 
> Literal.sql emits escaped SQL independently of the session's string-literal 
> parsing setting. Reading that SQL through the legacy parser mode retains 
> serialization escapes as value characters.
> Decode serialized literals using the escaping policy that Literal.sql emits, 
> including recursive literal-only constructors and collated strings. Share 
> this decoding with V2 column conversion while retaining its full analysis and 
> constant validation. Other session settings, including the time zone, remain 
> active. Expression-shaped external metadata, including casts containing 
> functions, keeps session parsing and existing reader evaluation behavior.
> h3. Affected versions and validation
> The reader path was introduced in Spark 3.4.0 (SPARK-39211 / SPARK-39862); 
> source-history inspection establishes the affected-version field. The V2 path 
> that reads EXISTS_DEFAULT dates to Spark 3.5.2 (SPARK-48286). Historical 
> releases were inspected statically; runtime validation was performed on the 
> PR checkout.
> 313 tests pass across StructTypeSuite, CatalogV2UtilSuite, 
> LiteralExpressionSuite, ExpressionParserSuite, InsertSuite and 
> CreateTableLikeSuite. Regressions cover both parser modes, schema 
> persistence, nested strings and collations, external CAST compatibility, V2 
> analysis, real Parquet reads and V1-to-V2 CREATE TABLE LIKE. Catalyst and SQL 
> Core production/test scalastyle checks pass.
> PR: https://github.com/apache/spark/pull/59820



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to