Szehon Ho created SPARK-59793:
---------------------------------
Summary: Preserve column default expressions in CTAS and RTAS
Key: SPARK-59793
URL: https://issues.apache.org/jira/browse/SPARK-59793
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
CTAS and RTAS inherit column default metadata from their input query, but do
not pass the available current-default connector expression to a Data Source V2
catalog. CREATE TABLE and REPLACE TABLE provide both SQL and a V2 expression
when translation is supported.
For example:
{code:sql}
CREATE TABLE catalog.src (i INT DEFAULT 42) USING provider;
CREATE TABLE catalog.dst USING provider AS SELECT * FROM catalog.src;
{code}
The default passed to the catalog for dst has SQL "42" and an
existence/backfill literal of 42, but getExpression() is null. This also
affects REPLACE TABLE AS SELECT and CREATE OR REPLACE TABLE AS SELECT,
including staging catalogs.
The proposed fix is scoped to the common CTAS/RTAS execution helper. It
translates the current-default expression already carried in the query schema
and passes it alongside the SQL text. It preserves the separate existence
default and retains SQL fallback when no translatable expression is available.
The shared schema-to-V2-column conversion is unchanged.
Regression coverage includes literal, arithmetic/cast, and scalar-function
defaults with expression folding both enabled and disabled, plus distinct
existence defaults and SQL fallback.
Pull request: https://github.com/apache/spark/pull/59066
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
