peter-toth commented on code in PR #59965:
URL: https://github.com/apache/spark/pull/59965#discussion_r4437178420


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -123,23 +132,47 @@ class ResolveTableConstraints(val catalogManager: 
CatalogManager) extends Rule[L
           val colRef = UnresolvedAttribute.quoted(col.name)
           val columnExtractors = Seq(col.name -> colRef)
           val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
-          CheckInvariant(
-            EqualNullSafe(colRef, catalystExpr),
+          def check(expr: Expression): Expression = CheckInvariant(
+            EqualNullSafe(colRef, expr),
             columnExtractors,
             "Generated Column",
             s"${col.name} <=> $genExprSql")
+          analyzeGeneratedColumnCheck(check(catalystExpr), 
v2Write.query).getOrElse {

Review Comment:
   **Finding 14.** A copy-on-write `DELETE` writes back the rows it keeps, and 
this check now reads their generation expression under the fixed settings. So 
rows that 4.3 auto-filled under an ANSI-off session fail it, also when the 
`DELETE` doesn't touch them. I ran this on base and head with ANSI off for the 
DDL and the `DELETE`, on `InMemoryRowLevelOperationTableCatalog`. I wrote the 
rows through the catalog, the way a 4.3 writer with ANSI off stores them:
   
   ```sql
   CREATE TABLE t (id INT, a STRING, b DOUBLE GENERATED ALWAYS AS (a + 1));
   -- stored rows: (1, 'abc', NULL), (2, '1.5', 2.5), (3, '7', 8.0)
   DELETE FROM t WHERE id >= 3;
   -- base: succeeds; head: CAST_INVALID_INPUT on the carried row 1
   ```
   
   The migration guide states this cost. But no config restores the 4.3 
reading, and a `DELETE` only works again after the table is recreated (finding 
17).
   
   A `DELETE` writes no new generated value, so there is nothing to check. The 
`WriteDelta` case at the top of `apply` already skips the checks for that 
reason, and `RewriteRowLevelCommand` rejects UPDATE and MERGE on such tables. 
Doing the same for a `ReplaceData`, right after the capability check in 
`buildGeneratedColumnConstraints`, fixes it:
   
   ```scala
       v2Write match {
         case w: RowLevelWrite if w.operation.command() == 
RowLevelOperation.Command.DELETE =>
           return Seq.empty
         case _ =>
       }
   ```
   
   With it the `DELETE` above passes on head. `GeneratedColumnWriteSuite`, 
`DataSourceV2DataFrameSuite`, the `DeleteFromTable*` suites and 
`CheckConstraintSuite` still pass (566 tests). The two DELETE tests here would 
then pass without reaching the check. A re-insert such as `INSERT INTO t SELECT 
* FROM t` goes through the check, so it could take over "a later write carries 
a row". The migration guide's sentence about a `DELETE` would go too.
   
   If you'd rather keep checking a `DELETE`, a `spark.sql.legacy.*` config that 
reads the text under the writer's session, as 4.3 does, is the other way out.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -4122,11 +4125,44 @@ class Analyzer(
               cleanedRelation
             case other => other
           }
-          v2Write.withNewQuery(projection).withNewTable(cleanedTable)
+          v2Write
+            .withNewQuery(analyzeAutoFilledGeneratedColumns(
+              projection, autoFilledGenCols, 
resolveOutput(CatalystSqlParser.parseExpression)._1))
+            .withNewTable(cleanedTable)
         } else {
           v2Write
         }
     }
+
+    /**
+     * Analyzes the auto-filled generated columns of a write under the 
settings their stored text
+     * is read under, so that the implicit casts and the functions of a 
generation expression do
+     * not follow the session that writes.
+     *
+     * A column these settings cannot resolve comes from `readInSession`, 
where the session reads
+     * its text, and the analysis of the write resolves it under the session. 
So does a text a
+     * catalog stores that analyzes to another plan.
+     */
+    private def analyzeAutoFilledGeneratedColumns(
+        projection: LogicalPlan,
+        autoFilled: Set[String],
+        readInSession: => LogicalPlan): LogicalPlan = projection match {
+      case p @ Project(projectList, child) if autoFilled.nonEmpty =>
+        lazy val sessionCols = readInSession.expressions.collect {
+          case c: NamedExpression => c.name -> c
+        }.toMap
+        p.copy(projectList = projectList.map {
+          case c if autoFilled.contains(c.name) =>
+            SQLConf.withExistingConf(StoredExpressionText.conf(conf)) {
+              execute(Project(Seq(c), LocalRelation(child.output)))

Review Comment:
   **Finding 16.** A text whose reading under the fixed settings throws during 
analysis, rather than staying unresolved, gets no fallback. That holds here and 
in `analyzeGeneratedColumnCheck` (`ResolveTableConstraints.scala:160`). CREATE 
doesn't catch it either. The string inside `IDENTIFIER` is only parsed during 
analysis, so the parse trees that `readsAsInSession` compares look the same. 
With ANSI on and `spark.sql.ansi.doubleQuotedIdentifiers=true` for the DDL and 
the writes, on `testcat`, on base and head:
   
   ```sql
   CREATE TABLE t (a INT, b INT GENERATED ALWAYS AS (IDENTIFIER('"a"') + 1));  
-- accepted on both
   INSERT INTO t (a) VALUES (1);   -- base: b = 2; head: PARSE_SYNTAX_ERROR
   INSERT INTO t VALUES (5, 6);    -- base: succeeds; head: PARSE_SYNTAX_ERROR
   ```
   
   The nested analysis parses `"a"` with double-quoted identifiers off. So on 
head no session can write the table, including the one that created it. 
a2ac9a984dd handles the same case when the parser throws, and this is that case 
at analysis time.
   
   Catching `AnalysisException` around the nested analysis in both places, and 
taking the session reading as for an unresolved result, fixes it. With it both 
inserts match base, and `GeneratedColumnWriteSuite` and 
`DataSourceV2DataFrameSuite` still pass. Rejecting it at CREATE instead would 
need `readsAsInSession` to parse the identifier strings too. A test with the 
two inserts above would pin either one.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -123,23 +132,47 @@ class ResolveTableConstraints(val catalogManager: 
CatalogManager) extends Rule[L
           val colRef = UnresolvedAttribute.quoted(col.name)
           val columnExtractors = Seq(col.name -> colRef)
           val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
-          CheckInvariant(
-            EqualNullSafe(colRef, catalystExpr),
+          def check(expr: Expression): Expression = CheckInvariant(
+            EqualNullSafe(colRef, expr),

Review Comment:
   **Finding 15.** Under the fixed settings, `<=>` coerces with the ANSI rules, 
so it casts the column side to the expression's type, not the other way round. 
The check then accepts any provided value that casts to the generated one, not 
only the value the auto-fill stores. I ran this on `testcat`, on base and head:
   
   ```sql
   CREATE TABLE t (s STRING, b STRING GENERATED ALWAYS AS (coalesce(s, 1)));
   INSERT INTO t (s) VALUES ('007');     -- head: b = '7'; base: '007' under an 
ANSI-off writer
   INSERT INTO t VALUES ('007', '07');   -- head: accepted with any writer
   INSERT INTO t VALUES ('007', ' 7');   -- head: accepted with any writer
   INSERT INTO t VALUES ('007', '007');  -- head: accepted with any writer
   INSERT INTO t VALUES ('007', 'abc');  -- head: CAST_INVALID_INPUT
   -- base, ANSI-off writer: CHECK_CONSTRAINT_VIOLATION for '7', '07', ' 7' and 
'abc'
   ```
   
   Head gives the same for each DDL and writer setting I tried. So the table 
ends up holding `'7'`, `'07'`, `' 7'` and `'007'` for the same `s`. Base does 
the same under an ANSI-on writer, so the default mode predates this PR. But now 
it also applies to every ANSI-off writer, where base fails loudly. Your [reply 
on finding 
5](https://github.com/apache/spark/pull/59965#discussion_r4436655467) notes the 
`'007'` case.
   
   Comparing with the value the auto-fill stores fixes it. Store assignment 
casts the expression to the column type (`TableOutputResolver.scala:973`), and 
the check can do the same (plus the `Cast` and `CharVarcharUtils` imports):
   
   ```scala
             def check(expr: Expression): Expression = CheckInvariant(
               EqualNullSafe(colRef, Cast(expr,
                 CharVarcharUtils.replaceCharVarcharWithString(col.dataType()),
                 Option(conf.sessionLocalTimeZone), ansiEnabled = true)),
   ```
   
   With it only `'7'` passes, for every writer, and `'abc'` fails with 
`CHECK_CONSTRAINT_VIOLATION`. `GeneratedColumnWriteSuite`, 
`DataSourceV2DataFrameSuite` and `CheckConstraintSuite` still pass. A test with 
the inserts above would pin it.
   



##########
docs/sql-migration-guide.md:
##########
@@ -69,6 +69,8 @@ license: |
 - Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with 
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to 
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in 
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a 
metadata column of the target table. Previously, these assignments passed 
analysis and were usually ignored. To restore the previous behavior, set 
`spark.sql.legacy.ignoreMetadataColumnAssignments` to `true`.
 - Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD 
CONSTRAINT` on a Data Source V2 table fail with 
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK 
constraint calls a user-defined function, including a Data Source V2 function. 
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with `UNSUPPORTED_SQL_UDF_USAGE` 
on a SQL function and with `INVALID_WHERE_CONDITION` on an aggregate function. 
Previously, a write could resolve the function name in its own session, so that 
it enforced another rule or failed (SPARK-60777). To run these statements, 
change the condition to call only built-in functions. Existing constraints that 
call a user-defined function are not rejected.
 - Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text 
of a CHECK constraint binds an unqualified built-in function name to 
`system.builtin`, regardless of the SQL path and 
`spark.sql.functionResolution.sessionOrder` (SPARK-60777). Previously, an 
existing constraint could call another function with the same name.
+- Since Spark 4.4, a write to a Data Source V2 table parses and analyzes the 
stored text of a generation expression with ANSI mode on and the default values 
of the SQL configs that change how a text reads, such as 
`spark.sql.parser.escapedStringLiterals`, regardless of the session of the 
write. Only the following still follow that session: the session time zone, 
case sensitivity, `spark.sql.timestampType`, 
`spark.sql.decimalOperations.allowPrecisionLoss`, the configs that a view also 
takes from the session that reads it, the functions found through `SET PATH`, 
and the configs that an expression reads during evaluation, such as 
`spark.sql.legacy.timeParserPolicy`. A text that does not parse or resolve 
under these settings, such as `flag = 1` on a `BOOLEAN` column, is read under 
that session. Previously, the session of the write decided how the text read, 
so a write could silently store a wrong value (SPARK-60757). As a result, with 
`spark.sql.ansi.enabled=false`, also for a table cre
 ated with that setting, a write can fail with an error such as 
`CAST_INVALID_INPUT` where Spark 4.3 stored `NULL`, and an implicit cast can 
change the stored value. A table created with a non-default value of a config 
such as `spark.sql.parser.escapedStringLiterals` also gets the default reading 
of its text on every write, without an error. A row that Spark 4.3 stored 
differently can fail with such an error or with `CHECK_CONSTRAINT_VIOLATION` 
when a later write, such as a `DELETE` that rewrites rows, carries it. To fix 
such a table, recreate the column with a text that reads as intended under 
these settings.

Review Comment:
   **Finding 17.** "To fix such a table, recreate the column with a text that 
reads as intended under these settings" can't be done in place:
   
   - `ALTER TABLE ... ADD COLUMN` doesn't accept `GENERATED ALWAYS AS` 
(`SqlBaseParser.g4:2018`).
   - `ALTER COLUMN` has no action that sets a generation expression 
(`SqlBaseParser.g4:2333`).
   
   So the fix is a new table with the corrected text, plus an `INSERT ... 
SELECT` of the stored columns from the old one. Saying that here would let a 
user act on it, for example:
   
   ```
   To fix such a table, create a new table with a text that reads as intended 
under these settings, and copy the other columns into it with `INSERT ... 
SELECT`.
   ```
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -123,23 +132,47 @@ class ResolveTableConstraints(val catalogManager: 
CatalogManager) extends Rule[L
           val colRef = UnresolvedAttribute.quoted(col.name)
           val columnExtractors = Seq(col.name -> colRef)
           val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
-          CheckInvariant(
-            EqualNullSafe(colRef, catalystExpr),
+          def check(expr: Expression): Expression = CheckInvariant(
+            EqualNullSafe(colRef, expr),
             columnExtractors,
             "Generated Column",
             s"${col.name} <=> $genExprSql")
+          analyzeGeneratedColumnCheck(check(catalystExpr), 
v2Write.query).getOrElse {
+            // The analysis of the write resolves it under the session that 
writes.
+            check(Option(genExpr.getSql).map(CatalystSqlParser.parseExpression)
+              .getOrElse(catalystExpr))
+          }
         }
     }.toSeq
   }
 
   /**
-   * Convert a V2 GenerationExpression to a Catalyst expression.
-   * Try the V2 expression first, fall back to parsing the SQL string.
+   * Analyzes the check of a provided value under the settings an auto-filled 
value is analyzed
+   * under, so that a row Spark auto-filled passes the check when a later 
write carries it.
+   *
+   * @return the analyzed check, or None if these settings cannot resolve it. 
A catalog can also
+   *         store a text that analyzes to another plan.
+   */
+  private def analyzeGeneratedColumnCheck(
+      check: Expression,
+      query: LogicalPlan): Option[Expression] = {
+    SQLConf.withExistingConf(StoredExpressionText.conf(conf)) {
+      analyze(Filter(check, LocalRelation(query.output)))
+    } match {
+      case Filter(analyzed, _: LocalRelation) if analyzed.resolved => 
Some(analyzed)
+      case _ => None
+    }
+  }
+
+  /**
+   * Convert a V2 GenerationExpression to a Catalyst expression. Read the SQL 
string first, as an
+   * auto-filled value does, and fall back to the V2 expression.
    */
   private def buildGenerationCatalystExpression(genExpr: 
GenerationExpression): Expression = {
-    Option(genExpr.getExpression)
-      .flatMap(V2ExpressionUtils.toCatalyst)
-      
.getOrElse(catalogManager.v1SessionCatalog.parser.parseExpression(genExpr.getSql))
+    Option(genExpr.getSql)
+      .map(StoredExpressionText.parse)
+      
.orElse(Option(genExpr.getExpression).flatMap(V2ExpressionUtils.toCatalyst))
+      .get

Review Comment:
   **Finding 21.** When the SQL string is null and 
`V2ExpressionUtils.toCatalyst` can't convert the V2 form, this throws 
`java.util.NoSuchElementException: None.get` without naming the column. The old 
code threw an NPE there. But `.get` makes the path look deliberate, so an 
`AnalysisException` that names the column and the expression would be clearer.
   



##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java:
##########
@@ -26,9 +26,9 @@
  * Connectors can define generated columns using either a SQL string (Spark 
SQL dialect) or a more
  * portable connector {@link Expression expression}, whose semantics follow 
the ANSI SQL standard.
  * <p>
- * If both the SQL string and the expression are provided, Spark first 
attempts to convert the
- * given expression to its internal representation. If the expression cannot 
be converted, and a
- * SQL string is provided, Spark will fall back to parsing the SQL string.
+ * If a SQL string is provided, Spark parses it, also when an expression is 
provided, because a

Review Comment:
   **Finding 23.** Two connector-facing details:
   
   - `Column.generationExpression()` (`Column.java:174`) still tells connectors 
to prefer `columnGenerationExpression()`, because it "captures the semantics 
unambiguously". Spark itself now reads the SQL string first.
   - "Otherwise, Spark converts the given expression" holds only for the check. 
Without a SQL string, `GeneratedColumn.attachGenerationExpressions` attaches 
nothing, so a write doesn't auto-fill the column.
   
   Saying here that Spark needs the SQL string to compute a value, and aligning 
`Column.java`, would cover both.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.catalyst.analysis.{Analyzer, 
ExpressionWithUnresolvedIdentifier}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, ExprId}
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Reads the SQL text of a generation expression stored with a table, 
independently of the
+ * settings of the session that reads it.
+ */
+object StoredExpressionText {
+
+  /**
+   * The settings a stored text is read under. The session time zone, case 
sensitivity, timestamp
+   * type and decimal precision loss stay with the given session, because the 
settings of the DDL
+   * are not known here and pinning them would silently change existing tables 
of a deployment that
+   * sets them.
+   */
+  def conf(session: SQLConf): SQLConf = {
+    val c = new SQLConf
+    Analyzer.retainResolutionConfigsForAnalysis(newConf = c, existingConf = 
session)
+    c.setConfString(SQLConf.ANSI_ENABLED.key, "true")
+    c.setConfString(SQLConf.SESSION_LOCAL_TIMEZONE.key, 
session.sessionLocalTimeZone)
+    c.setConfString(SQLConf.CASE_SENSITIVE.key, 
session.caseSensitiveAnalysis.toString)
+    c.setConf(SQLConf.TIMESTAMP_TYPE, session.getConf(SQLConf.TIMESTAMP_TYPE))
+    c.setConf(
+      SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS, 
session.decimalOperationsAllowPrecisionLoss)
+    c
+  }
+
+  /**
+   * Parses `text` under [[conf]]. A text these settings cannot parse is 
parsed under the active
+   * session, as a text they cannot resolve is analyzed under it.
+   */
+  def parse(text: String): Expression = {
+    try {
+      SQLConf.withExistingConf(conf(SQLConf.get)) {
+        CatalystSqlParser.parseExpression(text)
+      }
+    } catch {
+      case _: ParseException => CatalystSqlParser.parseExpression(text)
+    }
+  }
+
+  /**
+   * Whether `text` reads the same under [[conf]] as it does in the active 
session. The comparison
+   * uses the ANSI value of the session, so that it is about how the text 
reads.
+   */
+  def readsAsInSession(text: String): Boolean = {
+    val session = SQLConf.get
+    val fixed = conf(session)
+    fixed.setConfString(SQLConf.ANSI_ENABLED.key, session.ansiEnabled.toString)
+    val parsedInSession = CatalystSqlParser.parseExpression(text)
+    try {
+      val parsed = SQLConf.withExistingConf(fixed) {
+        CatalystSqlParser.parseExpression(text)
+      }
+      normalize(parsed) == normalize(parsedInSession)

Review Comment:
   **Finding 19.** With `spark.sql.ansi.enforceReservedKeywords=true`, this 
comparison also rejects a bare `CURRENT_DATE` or `CURRENT_TIMESTAMP` when no 
column has that name. The session parses it as the function, and the fixed 
settings as an attribute. A write then resolves that attribute to the same 
function. With ANSI on and that setting, on `testcat`, on base and head:
   
   ```sql
   CREATE TABLE t1 (a INT, d DATE GENERATED ALWAYS AS (current_date));          
 -- base: accepted; head: reads differently
   CREATE TABLE t2 (a INT, t TIMESTAMP GENERATED ALWAYS AS 
(CURRENT_TIMESTAMP));  -- base: accepted; head: reads differently
   ```
   
   So under that setting the standard spellings no longer work, and only 
`curdate()` and the like remain. The changed test in 
`DataSourceV2DataFrameSuite` covers a table with a column named `current_date`, 
where the rejection is needed. `validate` has `allColumns`, so it could accept 
a difference that is only a `current_*` keyword against an attribute that names 
no column. I haven't tried it.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala:
##########
@@ -124,6 +125,14 @@ case class GeneratedColumnExpression(
       throw unsupportedExpressionError(
         "generation expression cannot contain non utf8 binary collated string 
type")
     }
+
+    // A write reads the stored text under fixed settings, so that a value 
does not depend on the
+    // session that writes it. The text has to read the same there.
+    if (!StoredExpressionText.readsAsInSession(originalSQL)) {

Review Comment:
   **Finding 18.** `toV2` below (line 143) still builds the V2 form from the 
child as the DDL's session analyzed it. Every write now reads the text under 
the fixed settings. With ANSI off for the CREATE, on head:
   
   ```sql
   CREATE TABLE t (s STRING, b STRING GENERATED ALWAYS AS (coalesce(s, 1)));  
-- V2 form: COALESCE(s, '1')
   INSERT INTO t (s) VALUES ('007');  -- b = '7' with any writer
   ```
   
   So the catalog gets a V2 form that gives `'007'` for rows that Spark stores 
as `'7'`. Spark's own check no longer reads it after finding 5. But a connector 
that evaluates it disagrees with the stored rows. The open 
[SPARK-58269](https://issues.apache.org/jira/browse/SPARK-58269), which would 
infer partition filters from generated columns, would also have to pick one of 
two forms that disagree. On base the two agree whenever the DDL and the writer 
share the ANSI setting.
   
   Building the V2 form from the text analyzed under 
`StoredExpressionText.conf` would keep them together. So would leaving it null 
when that analysis differs from the child. I haven't tried either. [Finding 
12](https://github.com/apache/spark/pull/59965#discussion_r4435126218) would 
fix this by construction.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -149,7 +150,8 @@ object TableOutputResolver extends SQLConfHelper with 
Logging {
       byName: Boolean,
       conf: SQLConf,
       defaultValueFillMode: DefaultValueFillMode.Value,
-      ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+      ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+      readGenerationExpr: String => Expression = StoredExpressionText.parse

Review Comment:
   **Finding 22.** The defaults on the three private methods only serve the 
nested struct, array and map calls, which never see generation metadata. They 
also switch `resolveOutputColumns`, which the V1 insert paths use 
(`rules.scala:256` and `:559`), to the fixed parse without the fixed analysis. 
Dropping the defaults and passing the reader explicitly would make each caller 
choose.
   



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -426,7 +431,7 @@ object TableOutputResolver extends SQLConfHelper with 
Logging {
             // stored columns after this loop (see below), so the generated 
value is computed from
             // the values as they will be written. References to other 
generated columns are not
             // allowed.
-            val genExpr = CatalystSqlParser.parseExpression(genExprSql)
+            val genExpr = readGenerationExpr(genExprSql)

Review Comment:
   **Finding 20.** This predates the PR, but the comment above doesn't hold for 
a struct field. `resolveGenerationExprReferences` (line 841) replaces only 
single-part references. So `s.f` still resolves against the query's own column, 
before the store-assignment cast. On `testcat`, on base and head:
   
   ```sql
   CREATE TABLE t (id INT, s STRUCT<f: INT>, i INT,
     b INT GENERATED ALWAYS AS (s.f * 10), d INT GENERATED ALWAYS AS (i * 10));
   INSERT INTO t (id, s, i) SELECT 1, named_struct('f', 5.7D), 5.7D;
   -- stored: s.f = 5, i = 5, b = 57, d = 50
   ```
   
   So `b` doesn't match the stored `s`. The check of a later write that carries 
the row would compare it with `5 * 10`. I found no JIRA ticket for this. Could 
you file one?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to