Szehon Ho created SPARK-61495:
---------------------------------

             Summary: Use modeled DSv2 expressions for generated column writes
                 Key: SPARK-61495
                 URL: https://issues.apache.org/jira/browse/SPARK-61495
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Szehon Ho


Use modeled DSv2 expressions for generated column writes

GenerationExpression accepts either SQL text or a portable DSv2 expression. 
Spark can validate an explicitly supplied generated value using a modeled 
expression, but its auto-fill path currently reads only SQL text. The 
validation path also prefers SQL when both forms are present, while default 
values and CHECK constraints prefer the modeled form.

For a connector table declaring a INT and b INT with a modeled a + 1 expression 
and GENERATE_COLUMN_VALUES_ON_WRITE, inserting only a = 1 should store (1, 2). 
Today the missing b is treated as an ordinary missing column and the write 
fails its generated-column check.

Extend write resolution to convert generation expressions directly to Catalyst 
and compute them from stored input values after assignment casts. Share the 
conversion between auto-fill and validation. Prefer the modeled form, falling 
back to SQL only when the model is absent or unsupported by the converter. 
Report analysis and evaluation failures after conversion without changing 
representations.

This changes results when both representations exist and differ, including 
implicit casts preserved in the modeled expression by CREATE TABLE. For 
example, coalesce(s, 1) defined under a non-ANSI session generates '007' for s 
= '007' from its modeled string fallback, where reparsing SQL previously 
produced '7'.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to