dongjoon-hyun commented on code in PR #59965:
URL: https://github.com/apache/spark/pull/59965#discussion_r4452124779
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -878,7 +885,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
private def canCauseCastOverflow(cast: Cast): Boolean = {
containsIntegralOrDecimalType(cast.dataType) &&
- !Cast.canUpCast(cast.child.dataType, cast.dataType)
+ // The check of a generated column casts an expression before it is
resolved.
+ (!cast.child.resolved || !Cast.canUpCast(cast.child.dataType,
cast.dataType))
Review Comment:
Thank you. I reproduced both: a provided value failed with
`CAST_OVERFLOW_IN_TABLE_INSERT` for `a * 2` and for `a div 0`. In 7192b681ca7 I
took your second option and went back to the plain ANSI cast, and
`canCauseCastOverflow` and `TableOutputResolver.cast` are as on master again.
Your example now gives `ARITHMETIC_OVERFLOW` and `DIVIDE_BY_ZERO`, as on the
base. The test `SPARK-60757: the check of a provided value reports the error of
the expression` replaces the overflow test.
As you note, the only difference left is for a catalog-defined column whose
expression type doesn't up-cast to the column type: a provided value fails with
`CAST_OVERFLOW`, and an auto-filled one with `CAST_OVERFLOW_IN_TABLE_INSERT`.
##########
docs/sql-migration-guide.md:
##########
@@ -71,6 +71,8 @@ license: |
- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when
`hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies
user identities (any recognized type other than `NONE`/`NOSASL`), because
executor-side data access still runs as the server's own service identity
rather than the connecting user (SPARK-5159). Set
`spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the
limitation and silence the warning. Spark 5.0 is expected to refuse to start on
this configuration unless that option is set.
- Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a
metadata column of the target table. Previously, these assignments passed
analysis and were usually ignored. To restore the previous behavior, set
`spark.sql.legacy.ignoreMetadataColumnAssignments` to `true`.
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or failed (SPARK-60777). To run these statements, change the condition to call
only built-in functions. Existing constraints that call a user-defined function
are not rejected.
+- Since Spark 4.4, a write to a Data Source V2 table parses and analyzes the
stored text of a generation expression with ANSI mode on and the default values
of the SQL configs that change how a text reads, such as
`spark.sql.parser.escapedStringLiterals`, regardless of the session of the
write. Only the following still follow that session: the session time zone,
case sensitivity, `spark.sql.timestampType`,
`spark.sql.decimalOperations.allowPrecisionLoss`, the configs that a view also
takes from the session that reads it, the functions found through `SET PATH`,
and the configs that an expression reads during evaluation, such as
`spark.sql.legacy.timeParserPolicy`. Any other config that the parsing or the
analysis reads takes its default value, also one that a function reads while it
is built, such as `spark.sql.legacy.negativeIndexInArrayInsert`. A text that
does not resolve under these settings, such as `flag = 1` on a `BOOLEAN`
column, is parsed with the ANSI mode of that session a
nd analyzed under that session, and a text that does not parse under them is
parsed under that session. Previously, the session of the write decided how the
text read, so a write could silently store a wrong value (SPARK-60757). As a
result, with `spark.sql.ansi.enabled=false`, also for a table created with that
setting, a write can fail with an error such as `CAST_INVALID_INPUT` where
Spark 4.3 stored `NULL`, and an implicit cast can change the stored value. The
validation of a provided value compares it with the value that the write would
store, so a value that only casts to it now fails with
`CHECK_CONSTRAINT_VIOLATION`, also with ANSI mode on. For example, for `b
STRING GENERATED ALWAYS AS (coalesce(s, 1))` with `s = '007'`, which stores
`7`, a provided `07` fails. A table created with a non-default value of a
config such as `spark.sql.parser.escapedStringLiterals` also gets the default
reading of its text on every write, without an error. A table CHECK constraint
is still read
under the session that writes, so a constraint that repeats the text of a
generation expression can fail a write whose session sets such a config. A row
that Spark 4.3 stored differently can fail with such an error or with
`CHECK_CONSTRAINT_VIOLATION` when a later write provides it again, such as an
`INSERT ... SELECT` from the same table. A `DELETE` does not check the
generated columns of the rows that it carries. To fix such a table, create a
new table with a text that reads as intended under these settings, and copy the
other columns into it with `INSERT ... SELECT`. To restore the previous
behavior, set `spark.sql.legacy.useCurrentConfigsForGeneratedColumn` to `true`.
Review Comment:
Right, I confirmed that `07` still fails under an ANSI writer with the flag
on, where the base accepts it. In 7192b681ca7 the entry now says "To read the
text under the session of the write, as Spark 4.3 does, set ...". The PR
description lists the changes that stay with the flag on. The flag's test now
also inserts a provided value, which the check reads under the session too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
