Szehon Ho created SPARK-61806:
---------------------------------
Summary: Support generated column values in UPDATE and MERGE
Key: SPARK-61806
URL: https://issues.apache.org/jira/browse/SPARK-61806
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
DSv2 tables that advertise SUPPORTS_GENERATED_COLUMNS_ON_WRITE already allow
Spark to compute generated columns on INSERT. UPDATE and MERGE are currently
rejected when such a table has generated columns.
Extend the existing support to row-level writes:
* Recompute omitted generated values from the aligned, post-cast assignments
for each UPDATE or MERGE action.
* Preserve simultaneous SET semantics and validate explicitly supplied
generated values against the final row.
* Allow MERGE actions to mix generated and explicitly supplied values,
including star actions.
* Skip generated-column checks for delete delta records.
* Reuse the modeled generation-expression reader and SQL fallback used by
INSERT and validation.
For example, for a supported table t with columns a, b, and g GENERATED ALWAYS
AS (a * 2), starting from (a=1, b=3, g=2):
{code:sql}
UPDATE t SET a = b + 1, b = 100;
{code}
The resulting row should be (a=4, b=100, g=8). Currently the statement is
rejected as an unsupported table operation.
The implementation depends on SPARK-61805 so a failing generated expression in
an unselected MERGE action does not fail during optimization.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]