[ 
https://issues.apache.org/jira/browse/SPARK-61495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dongjoon Hyun reassigned SPARK-61495:
-------------------------------------

    Assignee: Szehon Ho

> Use modeled DSv2 expressions for generated column writes
> --------------------------------------------------------
>
>                 Key: SPARK-61495
>                 URL: https://issues.apache.org/jira/browse/SPARK-61495
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Szehon Ho
>            Assignee: Szehon Ho
>            Priority: Major
>              Labels: pull-request-available
>
> Use modeled DSv2 expressions for generated column writes
> GenerationExpression accepts either SQL text or a portable DSv2 expression. 
> Spark can validate an explicitly supplied generated value using a modeled 
> expression, but its auto-fill path currently reads only SQL text. The 
> validation path also prefers SQL when both forms are present, while default 
> values and CHECK constraints prefer the modeled form.
> For a connector table declaring a INT and b INT with a modeled a + 1 
> expression and GENERATE_COLUMN_VALUES_ON_WRITE, inserting only a = 1 should 
> store (1, 2). Today the missing b is treated as an ordinary missing column 
> and the write fails its generated-column check.
> Extend write resolution to convert generation expressions directly to 
> Catalyst and compute them from stored input values after assignment casts. 
> Share the conversion between auto-fill and validation. Prefer the modeled 
> form, falling back to SQL only when the model is absent or unsupported by the 
> converter. Report analysis and evaluation failures after conversion without 
> changing representations.
> This changes results when both representations exist and differ, including 
> implicit casts preserved in the modeled expression by CREATE TABLE. For 
> example, coalesce(s, 1) defined under a non-ANSI session generates '007' for 
> s = '007' from its modeled string fallback, where reparsing SQL previously 
> produced '7'.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to