szehon-ho opened a new pull request, #59066: URL: https://github.com/apache/spark/pull/59066
### What changes were proposed in this pull request? Preserve current-default expressions when converting query schemas to Data Source V2 columns for CTAS and RTAS. `CatalogV2Util.structTypeToV2Columns` now reads the expression stored alongside the SQL text in column metadata and translates it with `V2ExpressionBuilder`, as CREATE TABLE does. For example, after creating `src(i INT DEFAULT 42)`, `CREATE TABLE dst AS SELECT * FROM src` currently passes `ColumnDefaultValue(sql="42", expression=null, value=42)` to the connector. This change also supplies the V2 literal expression for `42`. The same fix applies to REPLACE TABLE AS SELECT and CREATE OR REPLACE TABLE AS SELECT, including staging catalogs. The existence/backfill literal remains separate. Defaults without a stored expression, or whose expression cannot be translated to V2, retain the SQL fallback. ### Why are the changes needed? CREATE TABLE provides connectors with both SQL and a V2 expression when translation is supported. CTAS and RTAS currently discard that expression during the schema-to-column conversion, even for literal defaults. Connectors then have to interpret Spark SQL to recover a default that was already available as a connector expression. ### Does this PR introduce _any_ user-facing change? Yes. V2 connectors receive the available current-default expression alongside its SQL text for inherited CTAS/RTAS defaults. SQL-only defaults and existence/backfill values retain their existing behavior. No public API changes are introduced. ### How was this patch tested? Added regression coverage in `CatalogV2UtilSuite` for distinct current and existence defaults and for SQL fallback with absent or unsupported expressions. Added end-to-end coverage in `DataSourceV2SQLSuiteV1Filter` for CTAS, RTAS, and CREATE OR REPLACE TABLE AS SELECT using both ordinary and staging catalogs, with literal, foldable, and SQL-only defaults. `git diff --check`, changed-file line-length checks, and non-ASCII checks passed. The new tests have not been run locally yet. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
