dongjoon-hyun commented on code in PR #60075:
URL: https://github.com/apache/spark/pull/60075#discussion_r4471098104
##########
docs/sql-migration-guide.md:
##########
@@ -74,6 +74,7 @@ license: |
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or failed (SPARK-60777). To run these statements, change the condition to call
only built-in functions. Existing constraints that call a user-defined function
are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and every later write to the table
failed (SPARK-60800). To run these statements, remove the function from the
condition. Existing constraints that call these functions are not rejected.
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a generator function, such as `explode`, `posexplode`, and
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded
and every later write to the table failed (SPARK-60821). To run these
statements, remove the function from the condition. Existing constraints that
call a generator function are not rejected.
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` fail with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when the data type of a generation
expression names its struct fields differently from the column type, in name,
including letter case, or in order, for example `named_struct('y', a, 'x', b)`,
`named_struct('X', b, 'Y', a)`, or `struct(a, b)` for a `STRUCT<x: INT, y:
INT>` column. Previously, such an expression was accepted, although Spark lines
up the struct fields by name, with the case sensitivity of the writing session,
when it computes the value of a generated column of a Data Source V2 table, and
by position when it checks a provided value (SPARK-60857). To line up the
fields by position, cast the expression to the column type, for example
`CAST(struct(a, b) AS STRUCT<x: INT, y: INT>)`. Existing tables with such an
expression are not rejected, and a write that carries a row that Spark
auto-filled into one of them keeps failing the check.
Review Comment:
Thank you for checking each shape. You're right, the sentence holds only for
fields of the same type in another order. I changed it in both the migration
guide and the description to say that existing tables are not rejected and that
this change does not affect writes to them. I used "does not affect" rather
than "behave as before" since SPARK-60757 also changed the check of a provided
value in 4.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
