dongjoon-hyun commented on code in PR #59965:
URL: https://github.com/apache/spark/pull/59965#discussion_r4437394565
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -426,7 +431,7 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
// stored columns after this loop (see below), so the generated
value is computed from
// the values as they will be written. References to other
generated columns are not
// allowed.
- val genExpr = CatalystSqlParser.parseExpression(genExprSql)
+ val genExpr = readGenerationExpr(genExprSql)
Review Comment:
Confirmed on both the base and this PR. It stores `b = 57` for a stored `s.f
= 5`, and re-inserting the row fails with `CHECK_CONSTRAINT_VIOLATION`. This
predates this PR, so I kept it out of this one. I am working on a fix
separately, and will file a JIRA ticket for it. Thank you for finding it.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -123,23 +132,47 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
val colRef = UnresolvedAttribute.quoted(col.name)
val columnExtractors = Seq(col.name -> colRef)
val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
- CheckInvariant(
- EqualNullSafe(colRef, catalystExpr),
+ def check(expr: Expression): Expression = CheckInvariant(
+ EqualNullSafe(colRef, expr),
columnExtractors,
"Generated Column",
s"${col.name} <=> $genExprSql")
+ analyzeGeneratedColumnCheck(check(catalystExpr),
v2Write.query).getOrElse {
+ // The analysis of the write resolves it under the session that
writes.
+ check(Option(genExpr.getSql).map(CatalystSqlParser.parseExpression)
+ .getOrElse(catalystExpr))
+ }
}
}.toSeq
}
/**
- * Convert a V2 GenerationExpression to a Catalyst expression.
- * Try the V2 expression first, fall back to parsing the SQL string.
+ * Analyzes the check of a provided value under the settings an auto-filled
value is analyzed
+ * under, so that a row Spark auto-filled passes the check when a later
write carries it.
+ *
+ * @return the analyzed check, or None if these settings cannot resolve it.
A catalog can also
+ * store a text that analyzes to another plan.
+ */
+ private def analyzeGeneratedColumnCheck(
+ check: Expression,
+ query: LogicalPlan): Option[Expression] = {
+ SQLConf.withExistingConf(StoredExpressionText.conf(conf)) {
+ analyze(Filter(check, LocalRelation(query.output)))
+ } match {
+ case Filter(analyzed, _: LocalRelation) if analyzed.resolved =>
Some(analyzed)
+ case _ => None
+ }
+ }
+
+ /**
+ * Convert a V2 GenerationExpression to a Catalyst expression. Read the SQL
string first, as an
+ * auto-filled value does, and fall back to the V2 expression.
*/
private def buildGenerationCatalystExpression(genExpr:
GenerationExpression): Expression = {
- Option(genExpr.getExpression)
- .flatMap(V2ExpressionUtils.toCatalyst)
-
.getOrElse(catalogManager.v1SessionCatalog.parser.parseExpression(genExpr.getSql))
+ Option(genExpr.getSql)
+ .map(StoredExpressionText.parse)
+
.orElse(Option(genExpr.getExpression).flatMap(V2ExpressionUtils.toCatalyst))
+ .get
Review Comment:
Fixed in 0fd998404af. The write now fails with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN`, which names the column and the
expression. The test `SPARK-60757: a V2 expression without a text that does not
convert names its column` covers it.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -149,7 +150,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
byName: Boolean,
conf: SQLConf,
defaultValueFillMode: DefaultValueFillMode.Value,
- ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+ ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+ readGenerationExpr: String => Expression = StoredExpressionText.parse
Review Comment:
Partly applied in 0fd998404af. `resolveOutputColumns` now passes the session
reader explicitly, so the V1 paths read as on the base, and
`resolveOutputColumnsWithGeneratedInfo` requires a reader. A V1 table cannot
get a generated column through SQL, since CREATE fails with
`UNSUPPORTED_FEATURE.TABLE_OPERATION`, so I did not add a test for it.
I kept the defaults on `reorderColumnsByName` and
`resolveColumnsByPosition`. Removing them means passing the reader through the
struct, array and map methods, which never use it.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java:
##########
@@ -26,9 +26,9 @@
* Connectors can define generated columns using either a SQL string (Spark
SQL dialect) or a more
* portable connector {@link Expression expression}, whose semantics follow
the ANSI SQL standard.
* <p>
- * If both the SQL string and the expression are provided, Spark first
attempts to convert the
- * given expression to its internal representation. If the expression cannot
be converted, and a
- * SQL string is provided, Spark will fall back to parsing the SQL string.
+ * If a SQL string is provided, Spark parses it, also when an expression is
provided, because a
Review Comment:
Applied in 0fd998404af. The `GenerationExpression` javadoc now says that
Spark computes a value only from the SQL string, and
`Column.generationExpression()` points to it instead of preferring the other
method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
