Steve Zhang created SPARK-61888:
-----------------------------------
Summary: Let pipe |> SET add columns that do not exist
Key: SPARK-61888
URL: https://issues.apache.org/jira/browse/SPARK-61888
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Steve Zhang
This tracks letting the SQL pipe operator {{|> SET}} add columns that do not
exist.
Today, a {{|> SET}} assignment to a column that does not exist in the input
table fails with {{UNRESOLVED_COLUMN}}, while {{|> EXTEND}} adds a second
column when one with that name already exists. A query that sets the same
column on inputs with different schemas, such as when normalizing data from
several sources, therefore has to know each input's schema. With this change,
such an assignment appends the column after the existing columns, as {{|>
EXTEND}} does. An assignment to an existing column still replaces it in place,
a name that matches more than one column still fails with
{{AMBIGUOUS_REFERENCE}}, and aggregate functions are still rejected.
{code:sql}
-- t has the columns x INT and y STRING.
TABLE t
|> SET z = x + length(y), x = x * 10, zz = z + x;
-- Output columns: x, y, z, zz. x is replaced in place; z and zz are appended.
{code}
The new behavior is on by default. Setting the internal config
{{spark.sql.pipeSetAddsMissingColumns.enabled}} to {{false}} restores the
previous error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]