Yan Yan created SPARK-61588:
-------------------------------

             Summary: Preserve persisted string defaults in legacy escaping mode
                 Key: SPARK-61588
                 URL: https://issues.apache.org/jira/browse/SPARK-61588
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.4.0
            Reporter: Yan Yan


With spark.sql.parser.escapedStringLiterals=true, Spark can reconstruct a 
persisted column default with extra backslashes. Rows written before ALTER 
TABLE ADD COLUMN can therefore return a different value from rows inserted 
later using DEFAULT.

h3. Reproduction

Run these statements in a fresh database. chr(39) constructs the apostrophe 
without depending on SQL string escaping in the reproduction itself.

{code:sql}
SET spark.sql.parser.escapedStringLiterals=true;
CREATE TABLE default_escape_repro (id INT) USING parquet;
INSERT INTO default_escape_repro VALUES (1);
ALTER TABLE default_escape_repro ADD COLUMNS (
  s STRING DEFAULT concat('Dianne', chr(39), 's horse')
);
INSERT INTO default_escape_repro VALUES (2, DEFAULT);
SELECT id, s, hex(s) FROM default_escape_repro ORDER BY id;
{code}

Expected: both rows contain {{Dianne's horse}}, with hex 
{{4469616E6E65277320686F727365}}.

Actual before the fix: the old row can contain {{Dianne\'s horse}}, with hex 
{{4469616E6E655C277320686F727365}}, while the new row contains the expected 
value.

Strings containing literal backslashes are also affected. The problem extends 
to strings nested in ARRAY, MAP and STRUCT defaults and to persisted default 
values converted for V2 catalogs, including CREATE TABLE LIKE.

h3. Cause and fix

Spark folds a default and stores its value as SQL in EXISTS_DEFAULT metadata. 
Literal.sql emits escaped SQL independently of the session's string-literal 
parsing setting. Reading that SQL through the legacy parser mode retains 
serialization escapes as value characters.

Decode serialized literals using the escaping policy that Literal.sql emits, 
including recursive literal-only constructors and collated strings. Share this 
decoding with V2 column conversion while retaining its full analysis and 
constant validation. Other session settings, including the time zone, remain 
active. Expression-shaped external metadata, including casts containing 
functions, keeps session parsing and existing reader evaluation behavior.

h3. Affected versions and validation

The reader path was introduced in Spark 3.4.0 (SPARK-39211 / SPARK-39862); 
source-history inspection establishes the affected-version field. The V2 path 
that reads EXISTS_DEFAULT dates to Spark 3.5.2 (SPARK-48286). Historical 
releases were inspected statically; runtime validation was performed on the PR 
checkout.

313 tests pass across StructTypeSuite, CatalogV2UtilSuite, 
LiteralExpressionSuite, ExpressionParserSuite, InsertSuite and 
CreateTableLikeSuite. Regressions cover both parser modes, schema persistence, 
nested strings and collations, external CAST compatibility, V2 analysis, real 
Parquet reads and V1-to-V2 CREATE TABLE LIKE. Catalyst and SQL Core 
production/test scalastyle checks pass.

PR: https://github.com/apache/spark/pull/59820




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to