dongjoon-hyun commented on code in PR #59930:
URL: https://github.com/apache/spark/pull/59930#discussion_r4419210949
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentUtils.scala:
##########
@@ -72,6 +72,8 @@ object AssignmentUtils extends SQLConfHelper with CastSupport
{
val errors = new mutable.ArrayBuffer[String]()
+ checkNoMetadataColumnAssignments(assignments, err => errors += err)
Review Comment:
Thank you. You are right that the alignment ran after
`ResolveSchemaEvolution` had altered the target table. I moved the check into a
new first case of `ResolveRowLevelCommandAssignments` that checks the keys as
soon as they are resolved, so the statement now fails before any schema change.
The new test `SPARK-60721: merge with schema evolution rejects assignments to
metadata columns before evolving the schema` covers `UPDATE SET *` with `INSERT
*` and an explicit assignment. Since the in-memory catalog is transactional,
the test checks the schema staged in the aborted transaction, which had the new
`bonus` column with the previous check.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -9028,6 +9028,18 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_IGNORE_METADATA_COLUMN_ASSIGNMENTS =
+ buildConf("spark.sql.legacy.ignoreMetadataColumnAssignments")
+ .internal()
+ .doc("When true, UPDATE and MERGE on a Data Source V2 table ignore
assignments to " +
Review Comment:
Thanks. I reworded the config doc and the migration guide: with the config,
Spark does not reject these assignments, as in Spark 4.3 and earlier, where
they passed analysis and were usually ignored. The comment on the check was
rewritten when the check moved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
