[
https://issues.apache.org/jira/browse/SPARK-61888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Zhang updated SPARK-61888:
--------------------------------
Description:
This tracks adding the SQL pipe operator {{|> EVAL}}, which replaces or appends
columns.
Today, a {{|> SET}} assignment to a column that does not exist in the input
table fails with {{UNRESOLVED_COLUMN}}, while {{|> EXTEND}} adds a second
column when one with that name already exists. A query that sets the same
column on inputs with different schemas, such as when normalizing data from
several sources, therefore has to know each input's schema. {{|> EVAL}} takes
the same assignments as {{|> SET}}. An assignment to a column that exists
replaces it in place, as {{SET}} does, and an assignment to a column that does
not exist appends it after the existing columns, as {{EXTEND}} does.
{code:sql}
-- t has the columns x INT and y STRING.
TABLE t
|> EVAL z = x + length(y), x = x * 10, zz = z + x;
-- Output columns: x, y, z, zz. x is replaced in place; z and zz are appended.
{code}
{{EVAL}} is a new non-reserved keyword. Existing operators, including {{SET}},
don't change.
was:
This tracks letting the SQL pipe operator {{|> SET}} add columns that do not
exist.
Today, a {{|> SET}} assignment to a column that does not exist in the input
table fails with {{UNRESOLVED_COLUMN}}, while {{|> EXTEND}} adds a second
column when one with that name already exists. A query that sets the same
column on inputs with different schemas, such as when normalizing data from
several sources, therefore has to know each input's schema. With this change,
such an assignment appends the column after the existing columns, as {{|>
EXTEND}} does. An assignment to an existing column still replaces it in place,
a name that matches more than one column still fails with
{{AMBIGUOUS_REFERENCE}}, and aggregate functions are still rejected.
{code:sql}
-- t has the columns x INT and y STRING.
TABLE t
|> SET z = x + length(y), x = x * 10, zz = z + x;
-- Output columns: x, y, z, zz. x is replaced in place; z and zz are appended.
{code}
The new behavior is on by default. Setting the internal config
{{spark.sql.pipeSetAddsMissingColumns.enabled}} to {{false}} restores the
previous error.
Summary: Add pipe |> EVAL to replace or append columns (was: Let pipe
|> SET add columns that do not exist)
> Add pipe |> EVAL to replace or append columns
> ---------------------------------------------
>
> Key: SPARK-61888
> URL: https://issues.apache.org/jira/browse/SPARK-61888
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Steve Zhang
> Priority: Major
> Labels: pull-request-available
>
> This tracks adding the SQL pipe operator {{|> EVAL}}, which replaces or
> appends columns.
> Today, a {{|> SET}} assignment to a column that does not exist in the input
> table fails with {{UNRESOLVED_COLUMN}}, while {{|> EXTEND}} adds a second
> column when one with that name already exists. A query that sets the same
> column on inputs with different schemas, such as when normalizing data from
> several sources, therefore has to know each input's schema. {{|> EVAL}} takes
> the same assignments as {{|> SET}}. An assignment to a column that exists
> replaces it in place, as {{SET}} does, and an assignment to a column that
> does not exist appends it after the existing columns, as {{EXTEND}} does.
> {code:sql}
> -- t has the columns x INT and y STRING.
> TABLE t
> |> EVAL z = x + length(y), x = x * 10, zz = z + x;
> -- Output columns: x, y, z, zz. x is replaced in place; z and zz are appended.
> {code}
> {{EVAL}} is a new non-reserved keyword. Existing operators, including
> {{SET}}, don't change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]