Steve Zhang created SPARK-61888:
-----------------------------------

             Summary: Let pipe |> SET add columns that do not exist
                 Key: SPARK-61888
                 URL: https://issues.apache.org/jira/browse/SPARK-61888
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Steve Zhang


This tracks letting the SQL pipe operator {{|> SET}} add columns that do not 
exist.

Today, a {{|> SET}} assignment to a column that does not exist in the input 
table fails with {{UNRESOLVED_COLUMN}}, while {{|> EXTEND}} adds a second 
column when one with that name already exists. A query that sets the same 
column on inputs with different schemas, such as when normalizing data from 
several sources, therefore has to know each input's schema. With this change, 
such an assignment appends the column after the existing columns, as {{|> 
EXTEND}} does. An assignment to an existing column still replaces it in place, 
a name that matches more than one column still fails with 
{{AMBIGUOUS_REFERENCE}}, and aggregate functions are still rejected.

{code:sql}
-- t has the columns x INT and y STRING.
TABLE t
|> SET z = x + length(y), x = x * 10, zz = z + x;
-- Output columns: x, y, z, zz. x is replaced in place; z and zz are appended.
{code}

The new behavior is on by default. Setting the internal config 
{{spark.sql.pipeSetAddsMissingColumns.enabled}} to {{false}} restores the 
previous error.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to