Yan Yan created SPARK-61588:
-------------------------------
Summary: Preserve persisted string defaults in legacy escaping mode
Key: SPARK-61588
URL: https://issues.apache.org/jira/browse/SPARK-61588
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.0
Reporter: Yan Yan
With spark.sql.parser.escapedStringLiterals=true, Spark can reconstruct a
persisted column default with extra backslashes. Rows written before ALTER
TABLE ADD COLUMN can therefore return a different value from rows inserted
later using DEFAULT.
h3. Reproduction
Run these statements in a fresh database. chr(39) constructs the apostrophe
without depending on SQL string escaping in the reproduction itself.
{code:sql}
SET spark.sql.parser.escapedStringLiterals=true;
CREATE TABLE default_escape_repro (id INT) USING parquet;
INSERT INTO default_escape_repro VALUES (1);
ALTER TABLE default_escape_repro ADD COLUMNS (
s STRING DEFAULT concat('Dianne', chr(39), 's horse')
);
INSERT INTO default_escape_repro VALUES (2, DEFAULT);
SELECT id, s, hex(s) FROM default_escape_repro ORDER BY id;
{code}
Expected: both rows contain {{Dianne's horse}}, with hex
{{4469616E6E65277320686F727365}}.
Actual before the fix: the old row can contain {{Dianne\'s horse}}, with hex
{{4469616E6E655C277320686F727365}}, while the new row contains the expected
value.
Strings containing literal backslashes are also affected. The problem extends
to strings nested in ARRAY, MAP and STRUCT defaults and to persisted default
values converted for V2 catalogs, including CREATE TABLE LIKE.
h3. Cause and fix
Spark folds a default and stores its value as SQL in EXISTS_DEFAULT metadata.
Literal.sql emits escaped SQL independently of the session's string-literal
parsing setting. Reading that SQL through the legacy parser mode retains
serialization escapes as value characters.
Decode serialized literals using the escaping policy that Literal.sql emits,
including recursive literal-only constructors and collated strings. Share this
decoding with V2 column conversion while retaining its full analysis and
constant validation. Other session settings, including the time zone, remain
active. Expression-shaped external metadata, including casts containing
functions, keeps session parsing and existing reader evaluation behavior.
h3. Affected versions and validation
The reader path was introduced in Spark 3.4.0 (SPARK-39211 / SPARK-39862);
source-history inspection establishes the affected-version field. The V2 path
that reads EXISTS_DEFAULT dates to Spark 3.5.2 (SPARK-48286). Historical
releases were inspected statically; runtime validation was performed on the PR
checkout.
313 tests pass across StructTypeSuite, CatalogV2UtilSuite,
LiteralExpressionSuite, ExpressionParserSuite, InsertSuite and
CreateTableLikeSuite. Regressions cover both parser modes, schema persistence,
nested strings and collations, external CAST compatibility, V2 analysis, real
Parquet reads and V1-to-V2 CREATE TABLE LIKE. Catalyst and SQL Core
production/test scalastyle checks pass.
PR: https://github.com/apache/spark/pull/59820
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]