peter-toth commented on code in PR #59965:
URL: https://github.com/apache/spark/pull/59965#discussion_r4453208371
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -119,27 +135,46 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
Option(col.columnGenerationExpression())
.filter(_ => !autoFilledGenCols.exists(n => resolver(n, col.name)))
.map { genExpr =>
- val catalystExpr = buildGenerationCatalystExpression(genExpr)
+ val catalystExpr = buildGenerationCatalystExpression(r, col.name,
genExpr)
val colRef = UnresolvedAttribute.quoted(col.name)
val columnExtractors = Seq(col.name -> colRef)
val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
- CheckInvariant(
- EqualNullSafe(colRef, catalystExpr),
+ // Compare with the value an auto-fill stores, which store
assignment casts to the column
+ // type. Otherwise, the comparison can cast the provided value to
the expression type.
+ val colType =
CharVarcharUtils.replaceCharVarcharWithString(col.dataType())
+ def check(expr: Expression): Expression = CheckInvariant(
+ EqualNullSafe(
+ colRef, Cast(expr, colType, Option(conf.sessionLocalTimeZone),
ansiEnabled = true)),
Review Comment:
**Finding 31.** The auto-fill and this check line up struct fields
differently. Store assignment matches the fields of an auto-filled struct to
the column type by name, and `Cast` matches them by position. `validate`
accepts such an expression. On `testcat`:
```sql
CREATE TABLE t (a INT, b INT,
s STRUCT<x: INT, y: INT> GENERATED ALWAYS AS (named_struct('y', a, 'x',
b)));
INSERT INTO t (a, b) VALUES (1, 2); -- stores s = {x: 2, y: 1}
INSERT INTO t SELECT * FROM t; -- CHECK_CONSTRAINT_VIOLATION
```
A DataFrame `writeTo(...).append()` stores the same shape, and a provided
`named_struct('x', 6, 'y', 5)` for `(5, 6)` fails too. Base gives the same
results, so this predates the PR, and I found no JIRA ticket for it. But the
comment above says that this compares with the value an auto-fill stores, and
this line is new, so it may be worth settling here. Two ways:
- `validate` could require the field names of a struct expression to match
the column type.
- This comparison could line the fields up by name, as store assignment does.
Otherwise, could you file a ticket?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4594,6 +4594,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val USE_CURRENT_SQL_CONFIGS_FOR_GENERATED_COLUMN =
+ buildConf("spark.sql.legacy.useCurrentConfigsForGeneratedColumn")
+ .internal()
+ .doc("When true, a write parses and analyzes the SQL text of a
generation expression of a " +
+ "Data Source V2 table under the SQL configs of the current active
SparkSession, and " +
+ "CREATE TABLE and REPLACE TABLE do not reject a text that the session
reads differently.")
+ .version("4.4.0")
Review Comment:
**Finding 30.** This entry has no `.withBindingPolicy(...)`, and
`sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions`
is a frozen list without this key. At this head, `build/sbt -Phive
'hive/testOnly *SparkConfigBindingPolicySuite'` fails "Config enforcement for
bindingPolicy", and it names only
`spark.sql.legacy.useCurrentConfigsForGeneratedColumn`. The
`ConfigBindingPolicy` scaladoc says a bug-fix flag shouldn't be `PERSISTED`,
and the sibling `spark.sql.legacy.ignoreMetadataColumnAssignments`
(`SQLConf.scala:9108`) uses `SESSION`. With the line below, the suite passes:
```suggestion
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.SESSION)
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala:
##########
@@ -119,27 +135,46 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
Option(col.columnGenerationExpression())
.filter(_ => !autoFilledGenCols.exists(n => resolver(n, col.name)))
.map { genExpr =>
- val catalystExpr = buildGenerationCatalystExpression(genExpr)
+ val catalystExpr = buildGenerationCatalystExpression(r, col.name,
genExpr)
val colRef = UnresolvedAttribute.quoted(col.name)
val columnExtractors = Seq(col.name -> colRef)
val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
- CheckInvariant(
- EqualNullSafe(colRef, catalystExpr),
+ // Compare with the value an auto-fill stores, which store
assignment casts to the column
+ // type. Otherwise, the comparison can cast the provided value to
the expression type.
+ val colType =
CharVarcharUtils.replaceCharVarcharWithString(col.dataType())
+ def check(expr: Expression): Expression = CheckInvariant(
+ EqualNullSafe(
+ colRef, Cast(expr, colType, Option(conf.sessionLocalTimeZone),
ansiEnabled = true)),
columnExtractors,
"Generated Column",
s"${col.name} <=> $genExprSql")
+ // Analyze the check under the settings an auto-filled value is
analyzed under, so that a
+ // row Spark auto-filled passes it when a later write carries it.
+ StoredExpressionText.analyze(
+ check(catalystExpr),
+
check(Option(genExpr.getSql).map(StoredExpressionText.parseWithSessionAnsi)
+ .getOrElse(catalystExpr)),
+ v2Write.query.output,
+ analyze)
}
}.toSeq
}
/**
- * Convert a V2 GenerationExpression to a Catalyst expression.
- * Try the V2 expression first, fall back to parsing the SQL string.
+ * Convert a V2 GenerationExpression to a Catalyst expression. Read the SQL
string first, as an
+ * auto-filled value does, and fall back to the V2 expression.
*/
- private def buildGenerationCatalystExpression(genExpr:
GenerationExpression): Expression = {
- Option(genExpr.getExpression)
- .flatMap(V2ExpressionUtils.toCatalyst)
-
.getOrElse(catalogManager.v1SessionCatalog.parser.parseExpression(genExpr.getSql))
+ private def buildGenerationCatalystExpression(
+ r: DataSourceV2Relation,
+ colName: String,
+ genExpr: GenerationExpression): Expression = {
+ Option(genExpr.getSql)
+ .map(StoredExpressionText.parse)
+
.orElse(Option(genExpr.getExpression).flatMap(V2ExpressionUtils.toCatalyst))
+ .getOrElse(throw QueryCompilationErrors.unsupportedTableOperationError(
+ r.catalog.get, r.identifier.get,
Review Comment:
**Finding 21.** This came back with the switch to
`unsupportedTableOperationError`
([r4437178453](https://github.com/apache/spark/pull/59965#discussion_r4437178453)
was the same `.get` shape). `DataFrameWriter.save()` on a `TableProvider`
builds the relation without a catalog or an identifier
(`DataFrameWriter.scala:186`, `:199`). So `r.catalog.get` throws here, and the
write fails with `NoSuchElementException: None.get` instead of this error. I
measured it with a provider that returns an `InMemoryTable` with the same
V2-only `NOT_A_FUNCTION()` column as the new test, written with
`df.write.format(...).mode("append").save()`. `checkNoGeneratedColumns`
(`RewriteRowLevelCommand.scala:67`) has the same `.get`, but it only sees
row-level commands, which always have a catalog.
Falling back to the table name works. With this, the probe gets ``Table
`probe_provider_table` does not support a value for the generated column `b`
...``, and the existing test still passes:
```scala
.getOrElse {
val operation = s"a value for the generated column
${toSQLId(colName)}, because " +
s"Spark cannot convert its expression ${genExpr.getExpression}"
throw ((r.catalog, r.identifier) match {
case (Some(catalog), Some(ident)) =>
QueryCompilationErrors.unsupportedTableOperationError(catalog,
ident, operation)
case _ =>
QueryCompilationErrors.unsupportedTableOperationError(r.table, operation)
})
}
```
plus a public overload next to `unsupportedBatchWriteError` in
`QueryCompilationErrors`:
```scala
def unsupportedTableOperationError(table: Table, operation: String):
Throwable = {
unsupportedTableOperationError(table.name(), operation)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
