This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new fb51de5959f5 [SPARK-60757][SQL] Do not apply a writer's session
settings to the generation expression of a generated column
fb51de5959f5 is described below
commit fb51de5959f5abbf40874bd6a2bd502a5676b133
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 21:13:20 2026 -0800
[SPARK-60757][SQL] Do not apply a writer's session settings to the
generation expression of a generated column
### What changes were proposed in this pull request?
This PR makes a write read the stored text of a generation expression under
fixed settings, instead of the settings of the session that writes.
- `StoredExpressionText` is a new helper that defines those settings: the
default `SQLConf`, the configs that
`Analyzer.retainResolutionConfigsForAnalysis` keeps, and ANSI enabled. The
session time zone, case sensitivity, `spark.sql.timestampType` and
`spark.sql.decimalOperations.allowPrecisionLoss` stay with the session, because
the settings of the DDL are not stored and pinning them would silently change
existing tables. Any other config that the parsing or the analysis reads takes
[...]
- `TableOutputResolver`, which auto-fills a generated column, and
`ResolveTableConstraints.buildGenerationCatalystExpression`, which validates a
provided value, parse the text under those settings with `CatalystSqlParser`.
The validation used to convert the V2 expression back first, which carries the
implicit casts of the session of the DDL. It now uses the V2 expression only
when there is no text, and the `GenerationExpression` and `Column` Javadocs say
so.
- `ResolveOutputRelation` analyzes the auto-filled columns under those
settings, and `ResolveTableConstraints` analyzes the check of a provided value
under them. Implicit casts and the ANSI behavior of functions are decided
during analysis, so parsing alone would leave them to the writer. Using the
same settings on both sides keeps a row that Spark auto-filled valid when a
later write carries it. The check also casts the expression to the column type
with an ANSI cast, as store assign [...]
- `ResolveTableConstraints` skips the check of generated columns for a
row-level `DELETE`, which writes no new generated value.
- `GeneratedColumnExpression.validate` rejects a generation expression that
reads differently under those settings than in the session that creates the
table, with a new reason of `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` that
names the parser configs. The comparison uses the ANSI value of the session, so
it is only about how the text reads.
- `spark.sql.legacy.useCurrentConfigsForGeneratedColumn` (internal, default
`false`, `SESSION` binding policy) reads the text under the session of the
write, as Apache Spark 4.3.0 does, and turns off the rejection above. The other
changes of this PR stay, such as reading the text before the V2 expression, the
cast in the check, and skipping a `DELETE`.
- `docs/sql-migration-guide.md` describes the behavior changes.
### Why are the changes needed?
Generated column values on V2 table writes were added by SPARK-57644 and
first released in Apache Spark
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27). The
value of a generated column depends on the session that writes it, and a wrong
value is stored silently.
```sql
CREATE TABLE t (a STRING, b STRING GENERATED ALWAYS AS (concat(a, 'x\\y')));
INSERT INTO t (a) VALUES ('default'); -- b = defaultx\y
SET spark.sql.parser.escapedStringLiterals=true;
INSERT INTO t (a) VALUES ('escaped'); -- b = escapedx\\y
INSERT INTO t (a, b) SELECT 'p', concat('p', 'x', chr(92), 'y'); --
CHECK_CONSTRAINT_VIOLATION
INSERT INTO t (a, b) SELECT 'q', concat('q', 'x', chr(92), chr(92), 'y');
-- succeeds
```
```sql
CREATE TABLE t (a STRING, b BIGINT GENERATED ALWAYS AS (a + 1));
SET spark.sql.ansi.enabled=false;
INSERT INTO t (a) VALUES ('abc'); -- b = NULL
INSERT INTO t (a) VALUES ('9007199254740993'); -- b = 9007199254740992
```
After this PR, the first example stores `escapedx\y`, accepts `p`, and
rejects `q`. The second example fails with `CAST_INVALID_INPUT` for `abc` and
stores `9007199254740994`.
### Does this PR introduce _any_ user-facing change?
Yes, compared to Apache Spark 4.3.0.
- The settings of the writer no longer change an auto-filled value or the
validation of a provided value through the parser settings, the ANSI mode, or a
temporary function that replaces a built-in function.
- A write with `spark.sql.ansi.enabled=false` now fails with the ANSI error
of a generation expression instead of storing `NULL`, also for a table created
with that setting. For example, `b DOUBLE GENERATED ALWAYS AS (a + 1)` with `a
= '1.5'` used to store `2.5` when both the DDL and the write ran with that
setting, and now fails with `CAST_INVALID_INPUT`. An implicit cast can also
change a stored value. For example, `b STRING GENERATED ALWAYS AS (coalesce(s,
1))` with `s = '007'` use [...]
- A table created with a non-default value of a config such as
`spark.sql.parser.escapedStringLiterals` gets the default reading of its text
on every write, without an error. So does a config that a function reads while
it is built. For example, with
`spark.sql.legacy.negativeIndexInArrayInsert=true`, `array_insert(a, -1, 'x')`
used to store `[a, x, b]` for `a = [a, b]`, and now stores `[a, b, x]`.
- A row that was stored that way can fail when a later write provides it
again, such as `INSERT INTO t SELECT * FROM t`. For example, with the `DOUBLE`
column above, a stored `b = 9.007199254740992E15` for `a = '9007199254740993'`
fails with `CHECK_CONSTRAINT_VIOLATION`, and a stored `b = NULL` for `a =
'abc'` fails with `CAST_INVALID_INPUT`. Such a write already fails the same way
in Apache Spark 4.3.0 when it runs with ANSI enabled.
- A `DELETE` no longer checks the rows that it carries, so a row that was
stored that way does not fail it. In Apache Spark 4.3.0, a `DELETE` that
rewrites rows fails on such a row when it runs with ANSI enabled.
- The validation of a provided value compares it with the value that the
write would store, so a value that only casts to it now fails with
`CHECK_CONSTRAINT_VIOLATION`, also with ANSI enabled. For example, for `b
STRING GENERATED ALWAYS AS (coalesce(s, 1))` with `s = '007'`, which stores
`7`, a provided `07` used to pass with ANSI enabled and now fails.
- A table CHECK constraint is still read under the session that writes. So
a constraint that repeats the text of a generation expression can fail a write
with a non-default parser setting, such as
`spark.sql.parser.escapedStringLiterals=true`, where Apache Spark 4.3.0 stored
the value that the setting gives.
- The V2 expression that `CREATE TABLE` gives a catalog still comes from
the analysis in the session of the DDL. With `spark.sql.ansi.enabled=false`, it
can differ from a value that a write stores. For example, it is `COALESCE(s,
'1')` for the `coalesce(s, 1)` column above. Spark itself uses it only when
there is no text.
- `CREATE TABLE` and `REPLACE TABLE` reject a generation expression that
the session reads differently. For example, with
`spark.sql.parser.escapedStringLiterals=true`, `concat(a, 'x\\y')` is rejected,
while `concat(a, 'xy')` is accepted. With
`spark.sql.ansi.enforceReservedKeywords=true`, a bare `current_date` is
rejected, and `curdate()` is accepted. This also applies to a table without the
`GENERATE_COLUMN_VALUES_ON_WRITE` capability.
- `spark.sql.legacy.useCurrentConfigsForGeneratedColumn=true` reads the
text under the session of the write, as Apache Spark 4.3.0 does, and restores
the previous DDL behavior.
The following still depend on the session that writes:
- A generation expression that the fixed settings cannot resolve or
analyze. It keeps their parser settings, and its ANSI behavior and analysis
follow the session as in Apache Spark 4.3.0. A text that they cannot parse is
parsed under the session. For example, `flag = 1` on a `BOOLEAN` column works
with `spark.sql.ansi.enabled=false` and fails with
`DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES` with ANSI enabled. `CAST(d AS
BOOLEAN)` on a `DATE` column stores `NULL` with ANSI disabled and f [...]
- The functions found through `SET PATH`.
- The session time zone, for example for `hour(ts)`.
- `spark.sql.timestampType`, for example for `to_timestamp(a)` and `CAST(a
AS TIMESTAMP)`.
- `spark.sql.decimalOperations.allowPrecisionLoss`, for example for `a * a`
on `DECIMAL` columns.
- An ANSI cast that `ConstantFolding` removes under a session with ANSI
disabled, such as in `size(array(CAST(a AS INT)))`, as in Apache Spark 4.3.0.
- The settings that an expression reads during evaluation, such as
`spark.sql.legacy.timeParserPolicy` and
`spark.sql.legacy.castComplexTypesToString.enabled`.
### How was this patch tested?
Added test cases to `GeneratedColumnWriteSuite`, updated one test case in
`DataSourceV2DataFrameSuite`, and changed the placeholder text in
`GeneratedColumnExpressionSuite`. Eighteen of the new test cases fail without
this PR.
Ran locally:
```bash
build/sbt 'catalyst/testOnly *GeneratedColumnExpressionSuite'
build/sbt 'sql/testOnly *GeneratedColumnWriteSuite
*DataSourceV2DataFrameSuite *CheckConstraintSuite *DeleteFromTable*'
SPARK_ANSI_SQL_MODE=false build/sbt 'sql/testOnly
*GeneratedColumnWriteSuite *DataSourceV2DataFrameSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This patch had conflicts when merged, resolved by
Committer: Dongjoon Hyun <[email protected]>
Closes #59965 from dongjoon-hyun/SPARK-60757.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 6e287bf9bf3bb6c6e464c53350db507b210dfd1e)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 2 +
.../apache/spark/sql/connector/catalog/Column.java | 8 +-
.../connector/catalog/GenerationExpression.java | 7 +-
.../spark/sql/catalyst/analysis/Analyzer.scala | 42 +-
.../analysis/ResolveTableConstraints.scala | 67 ++-
.../catalyst/analysis/TableOutputResolver.scala | 27 +-
.../plans/logical/GeneratedColumnExpression.scala | 13 +-
.../sql/catalyst/util/StoredExpressionText.scala | 142 +++++
.../spark/sql/errors/QueryCompilationErrors.scala | 4 +
.../org/apache/spark/sql/internal/SQLConf.scala | 14 +
.../logical/GeneratedColumnExpressionSuite.scala | 2 +-
.../sql/connector/DataSourceV2DataFrameSuite.scala | 37 +-
.../sql/connector/GeneratedColumnWriteSuite.scala | 572 ++++++++++++++++++++-
13 files changed, 882 insertions(+), 55 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index edb882b73634..2440185e1653 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -69,6 +69,8 @@ license: |
- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when
`hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies
user identities (any recognized type other than `NONE`/`NOSASL`), because
executor-side data access still runs as the server's own service identity
rather than the connecting user (SPARK-5159). Set
`spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the
limitation and silence the warning. Spark 5.0 is expected to [...]
- Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a
metadata column of the target table. Previously, these assignments passed
analysis and were usually ignored. To restore the previous behavior, set
`spark.sql.legacy.ignoreMetadata [...]
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or [...]
+- Since Spark 4.4, a write to a Data Source V2 table parses and analyzes the
stored text of a generation expression with ANSI mode on and the default values
of the SQL configs that change how a text reads, such as
`spark.sql.parser.escapedStringLiterals`, regardless of the session of the
write. Only the following still follow that session: the session time zone,
case sensitivity, `spark.sql.timestampType`,
`spark.sql.decimalOperations.allowPrecisionLoss`, the configs that a view also
tak [...]
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` fail with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` on a generation expression that the
session reads differently than a write does, for example with
`spark.sql.parser.escapedStringLiterals=true` or
`spark.sql.ansi.enforceReservedKeywords=true`. This also applies to a table
without the `GENERATE_COLUMN_VALUES_ON_WRITE` capability, which computes the
values itself (SPARK-60757). To restore the previous behavior, set
`spark.sql.legacy.u [...]
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and [...]
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a generator function, such as `explode`, `posexplode`, and
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded
and every later write to the table failed (SPARK-60821). To run these
statements, remove the function from the condition. E [...]
- Since Spark 4.4, `ALTER TABLE ... ADD CONSTRAINT` with a CHECK constraint
fails with `EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE` when the target is a view, as
it does with the other constraint types. Previously, when the view only
projected or filtered a Data Source V2 table, the statement added the
constraint to that table after validating only the rows of the view
(SPARK-60822). To add the constraint, run the statement on the table.
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java
index 438b64dbaf21..bf37787b5082 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java
@@ -171,10 +171,10 @@ public interface Column {
* Returns the generation expression of this table column as a SQL string.
Null means no
* generation expression.
* <p>
- * This returns only the SQL string form. Prefer {@link
#columnGenerationExpression()}, which can
- * also carry a connector {@link
org.apache.spark.sql.connector.expressions.Expression} and
- * captures the semantics unambiguously. It is up to the data source to
verify expression
- * compatibility and reject writes as necessary.
+ * This returns only the SQL string form. {@link
#columnGenerationExpression()} can also carry a
+ * connector {@link org.apache.spark.sql.connector.expressions.Expression}.
See
+ * {@link GenerationExpression} for how Spark reads each form. It is up to
the data source to
+ * verify expression compatibility and reject writes as necessary.
*/
@Nullable
default String generationExpression() {
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java
index 8e64eba9c516..0395620f53d6 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java
@@ -26,9 +26,10 @@ import org.apache.spark.sql.connector.expressions.Expression;
* Connectors can define generated columns using either a SQL string (Spark
SQL dialect) or a more
* portable connector {@link Expression expression}, whose semantics follow
the ANSI SQL standard.
* <p>
- * If both the SQL string and the expression are provided, Spark first
attempts to convert the
- * given expression to its internal representation. If the expression cannot
be converted, and a
- * SQL string is provided, Spark will fall back to parsing the SQL string.
+ * Spark computes a generated value on a write only from the SQL string, so a
column without one is
+ * not auto-filled. To check a provided value, Spark parses the SQL string if
one is provided, also
+ * when an expression is provided. Otherwise, Spark converts the given
expression to its internal
+ * representation.
*
* @since 4.3.0
*/
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index ef6d53cc3a9c..4ffa9c1d8117 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -48,7 +48,7 @@ import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.catalyst.trees.CurrentOrigin.withOrigin
import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.catalyst.types.DataTypeUtils
-import org.apache.spark.sql.catalyst.util.{toPrettySQL,
trimTempResolvedColumn, CharVarcharUtils, GeneratedColumn}
+import org.apache.spark.sql.catalyst.util.{toPrettySQL,
trimTempResolvedColumn, CharVarcharUtils, GeneratedColumn, StoredExpressionText}
import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns._
// `View` is aliased to `V2View` to avoid clashing with the logical-plan
`View` imported via
// `org.apache.spark.sql.catalyst.plans.logical._`.
@@ -715,7 +715,7 @@ class Analyzer(
ResolveWindowFrame ::
ResolveNaturalAndUsingJoin ::
ResolveOutputRelation ::
- new ResolveTableConstraints(catalogManager) ::
+ new ResolveTableConstraints(catalogManager, plan => execute(plan)) ::
new ResolveSetVariable(catalogManager) ::
new ResolveFetchCursor(catalogManager) ::
new ResolveCursors() ::
@@ -4123,10 +4123,12 @@ class Analyzer(
table.schemaAlignmentConfig()
case _ => SchemaAlignmentConfig.DEFAULT
}
- val (projection, autoFilledGenCols) =
+ def resolveOutput(readGenerationExpr: String => Expression) =
TableOutputResolver.resolveOutputColumnsWithGeneratedInfo(
v2Write.table.name, expected, v2Write.query, v2Write.isByName,
conf,
- defaultValueFillMode,
schemaAlignmentConfig.ansiStoreAssignmentCastCheck())
+ defaultValueFillMode,
schemaAlignmentConfig.ansiStoreAssignmentCastCheck(),
+ readGenerationExpr)
+ val (projection, autoFilledGenCols) =
resolveOutput(StoredExpressionText.parse)
if (projection != v2Write.query) {
val cleanedTable = v2Write.table match {
case r: DataSourceV2Relation =>
@@ -4137,11 +4139,41 @@ class Analyzer(
cleanedRelation
case other => other
}
- v2Write.withNewQuery(projection).withNewTable(cleanedTable)
+ v2Write
+ .withNewQuery(analyzeAutoFilledGeneratedColumns(
+ projection,
+ autoFilledGenCols,
+ resolveOutput(StoredExpressionText.parseWithSessionAnsi)._1))
+ .withNewTable(cleanedTable)
} else {
v2Write
}
}
+
+ /**
+ * Analyzes the auto-filled generated columns of a write under the
settings their stored text
+ * is read under, so that the implicit casts and the functions of a
generation expression do
+ * not follow the session that writes.
+ *
+ * A column these settings cannot resolve comes from `readInSession`,
where its text is read
+ * with the ANSI value of the session, and the analysis of the write
resolves it under the
+ * session. So does a text a catalog stores that analyzes to another plan.
+ */
+ private def analyzeAutoFilledGeneratedColumns(
+ projection: LogicalPlan,
+ autoFilled: Set[String],
+ readInSession: => LogicalPlan): LogicalPlan = projection match {
+ case p @ Project(projectList, child) if autoFilled.nonEmpty =>
+ lazy val sessionCols = readInSession.expressions.collect {
+ case c: NamedExpression => c.name -> c
+ }.toMap
+ p.copy(projectList = projectList.map {
+ case c if autoFilled.contains(c.name) =>
+ StoredExpressionText.analyze(c, sessionCols(c.name), child.output,
execute)
+ case c => c
+ })
+ case other => other
+ }
}
private def validateStoreAssignmentPolicy(): Unit = {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
index d07a3e9122bd..b961c5adfbcd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
@@ -19,17 +19,26 @@ package org.apache.spark.sql.catalyst.analysis
import scala.collection.mutable
import org.apache.spark.sql.catalyst.FunctionIdentifier
-import org.apache.spark.sql.catalyst.expressions.{And, CheckInvariant,
EqualNullSafe, Expression, V2ExpressionUtils}
-import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
V2WriteCommand, WriteDelta}
+import org.apache.spark.sql.catalyst.expressions.{And, Cast, CheckInvariant,
EqualNullSafe, Expression, V2ExpressionUtils}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
RowLevelWrite, V2WriteCommand, WriteDelta}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.TreePattern.{COMMAND,
LAMBDA_VARIABLE}
-import org.apache.spark.sql.catalyst.util.GeneratedColumn
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, GeneratedColumn,
StoredExpressionText}
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.connector.catalog.{CatalogManager,
GenerationExpression}
import org.apache.spark.sql.connector.catalog.constraints.Check
import org.apache.spark.sql.connector.write.RowLevelOperation
+import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-class ResolveTableConstraints(val catalogManager: CatalogManager) extends
Rule[LogicalPlan] {
+/**
+ * @param analyze Analyzes a plan under the active SQLConf. The check of a
generated column goes
+ * through it, so that the settings its stored text is read
under cover its analysis
+ * without applying to the query of the write.
+ */
+class ResolveTableConstraints(
+ val catalogManager: CatalogManager,
+ analyze: LogicalPlan => LogicalPlan) extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveOperatorsWithPruning(
_.containsPattern(COMMAND), ruleId) {
@@ -108,6 +117,13 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
return Seq.empty
}
+ // A DELETE writes no new generated value, so the rows it carries have
nothing to check.
+ v2Write match {
+ case w: RowLevelWrite if w.operation.command() ==
RowLevelOperation.Command.DELETE =>
+ return Seq.empty
+ case _ =>
+ }
+
val resolver = catalogManager.v1SessionCatalog.conf.resolver
val autoFilledGenCols = r.output
.filter(GeneratedColumn.isAutoFilledGeneratedColumn)
@@ -119,27 +135,52 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
Option(col.columnGenerationExpression())
.filter(_ => !autoFilledGenCols.exists(n => resolver(n, col.name)))
.map { genExpr =>
- val catalystExpr = buildGenerationCatalystExpression(genExpr)
+ val catalystExpr = buildGenerationCatalystExpression(r, col.name,
genExpr)
val colRef = UnresolvedAttribute.quoted(col.name)
val columnExtractors = Seq(col.name -> colRef)
val genExprSql = Option(genExpr.getSql()).getOrElse(catalystExpr.sql)
- CheckInvariant(
- EqualNullSafe(colRef, catalystExpr),
+ // Compare with the value an auto-fill stores, which store
assignment casts to the column
+ // type. Otherwise, the comparison can cast the provided value to
the expression type.
+ val colType =
CharVarcharUtils.replaceCharVarcharWithString(col.dataType())
+ def check(expr: Expression): Expression = CheckInvariant(
+ EqualNullSafe(
+ colRef, Cast(expr, colType, Option(conf.sessionLocalTimeZone),
ansiEnabled = true)),
columnExtractors,
"Generated Column",
s"${col.name} <=> $genExprSql")
+ // Analyze the check under the settings an auto-filled value is
analyzed under, so that a
+ // row Spark auto-filled passes it when a later write carries it.
+ StoredExpressionText.analyze(
+ check(catalystExpr),
+
check(Option(genExpr.getSql).map(StoredExpressionText.parseWithSessionAnsi)
+ .getOrElse(catalystExpr)),
+ v2Write.query.output,
+ analyze)
}
}.toSeq
}
/**
- * Convert a V2 GenerationExpression to a Catalyst expression.
- * Try the V2 expression first, fall back to parsing the SQL string.
+ * Convert a V2 GenerationExpression to a Catalyst expression. Read the SQL
string first, as an
+ * auto-filled value does, and fall back to the V2 expression.
*/
- private def buildGenerationCatalystExpression(genExpr:
GenerationExpression): Expression = {
- Option(genExpr.getExpression)
- .flatMap(V2ExpressionUtils.toCatalyst)
-
.getOrElse(catalogManager.v1SessionCatalog.parser.parseExpression(genExpr.getSql))
+ private def buildGenerationCatalystExpression(
+ r: DataSourceV2Relation,
+ colName: String,
+ genExpr: GenerationExpression): Expression = {
+ Option(genExpr.getSql)
+ .map(StoredExpressionText.parse)
+
.orElse(Option(genExpr.getExpression).flatMap(V2ExpressionUtils.toCatalyst))
+ .getOrElse {
+ val operation = s"a value for the generated column
${toSQLId(colName)}, because " +
+ s"Spark cannot convert its expression ${genExpr.getExpression}"
+ // A relation of a TableProvider has no catalog and no identifier.
+ throw ((r.catalog, r.identifier) match {
+ case (Some(catalog), Some(ident)) =>
+ QueryCompilationErrors.unsupportedTableOperationError(catalog,
ident, operation)
+ case _ =>
QueryCompilationErrors.unsupportedTableOperationError(r.table, operation)
+ })
+ }
}
private def buildColumnExtractors(
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
index 82f15532c542..c58d1a1dfe57 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
@@ -119,10 +119,11 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
AnsiStoreAssignmentCastCheck.DEFAULT): LogicalPlan = {
resolveOutputColumnsInternal(
tableName, expected, query, byName, conf, defaultValueFillMode,
- ansiStoreAssignmentCastCheck)._1
+ ansiStoreAssignmentCastCheck, CatalystSqlParser.parseExpression)._1
}
/**
+ * @param readGenerationExpr reads the stored text of a generation
expression.
* @return a tuple of:
* - the resolved write plan (same as [[resolveOutputColumns]]); and
* - the names of the generated columns that were auto-filled, i.e.
computed from their
@@ -135,11 +136,12 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
byName: Boolean,
conf: SQLConf,
defaultValueFillMode: DefaultValueFillMode.Value = NONE,
- ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+ ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+ readGenerationExpr: String => Expression
): (LogicalPlan, Set[String]) = {
resolveOutputColumnsInternal(
tableName, expected, query, byName, conf, defaultValueFillMode,
- ansiStoreAssignmentCastCheck)
+ ansiStoreAssignmentCastCheck, readGenerationExpr)
}
private def resolveOutputColumnsInternal(
@@ -149,7 +151,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
byName: Boolean,
conf: SQLConf,
defaultValueFillMode: DefaultValueFillMode.Value,
- ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+ ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+ readGenerationExpr: String => Expression
): (LogicalPlan, Set[String]) = {
if (expected.size < query.output.size) {
@@ -176,7 +179,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
Nil,
defaultValueFillMode,
enforceFullOutput = true,
- ansiStoreAssignmentCastCheck)
+ ansiStoreAssignmentCastCheck,
+ readGenerationExpr)
} else {
if (expected.size > query.output.size && !fillDefaultValue) {
throw QueryCompilationErrors.cannotWriteNotEnoughColumnsToTableError(
@@ -185,7 +189,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
resolveColumnsByPosition(
tableName, query.output, expected, conf, errors += _,
fillDefaultValue = fillDefaultValue,
- ansiStoreAssignmentCastCheck = ansiStoreAssignmentCastCheck)
+ ansiStoreAssignmentCastCheck = ansiStoreAssignmentCastCheck,
+ readGenerationExpr = readGenerationExpr)
}
if (errors.nonEmpty) {
@@ -415,7 +420,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
colPath: Seq[String] = Nil,
defaultValueFillMode: DefaultValueFillMode.Value,
enforceFullOutput: Boolean = false,
- ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+ ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+ readGenerationExpr: String => Expression =
CatalystSqlParser.parseExpression
): (Seq[NamedExpression], Set[String]) = {
// Names of generated columns that were auto-filled (not provided by the
user). Only populated
// for top-level columns, since generated columns cannot be nested.
@@ -433,7 +439,7 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
// The references are left unresolved here.
`resolveOutputColumnsInternal` resolves them
// against the stored columns, so the generated value is computed
from the values as
// they will be written. References to other generated columns are
not allowed.
- val genExpr = CatalystSqlParser.parseExpression(genExprSql)
+ val genExpr = readGenerationExpr(genExprSql)
Some(applyColumnMetadata(genExpr, expectedCol))
case None =>
val useNullAsDefault =
useNullAsDefaultForMissingColumn(expectedCols, conf)
@@ -518,7 +524,8 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
addError: String => Unit,
colPath: Seq[String] = Nil,
fillDefaultValue: Boolean = false,
- ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck
+ ansiStoreAssignmentCastCheck: AnsiStoreAssignmentCastCheck,
+ readGenerationExpr: String => Expression =
CatalystSqlParser.parseExpression
): (Seq[NamedExpression], Set[String]) = {
// Names of generated columns that were auto-filled (not provided by the
user). Only populated
// for top-level columns, since generated columns cannot be nested.
@@ -592,7 +599,7 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
autoFilledGenCols += expectedCol.name
// The references are resolved as in the by-name path. The stored
columns have the
// names of the table columns, even when the query uses different
names.
- val genExpr = CatalystSqlParser.parseExpression(genExprSql)
+ val genExpr = readGenerationExpr(genExprSql)
applyColumnMetadata(genExpr, expectedCol)
case None =>
val useNullAsDefault =
useNullAsDefaultForMissingColumn(actualExpectedCols, conf)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
index 19f1336559b2..9a9ad6d713ae 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.plans.logical
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.expressions.{AnalysisAwareExpression,
AttributeReference, Cast, Expression, UnaryExpression, Unevaluable}
import
org.apache.spark.sql.catalyst.trees.TreePattern.{ANALYSIS_AWARE_EXPRESSION,
PLAN_EXPRESSION, TreePattern, VARIABLE_REFERENCE}
-import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
+import org.apache.spark.sql.catalyst.util.{StoredExpressionText,
V2ExpressionBuilder}
import org.apache.spark.sql.connector.catalog.GenerationExpression
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DataType, TimeType}
@@ -59,6 +59,7 @@ case class GeneratedColumnExpression(
* - No subquery expressions
* - No references to session (temporary) variables
* - No non-UTF8 binary collation
+ * - The text reads the same as it does under the settings a write reads it
with
*/
def validate(
fieldName: String,
@@ -124,6 +125,16 @@ case class GeneratedColumnExpression(
throw unsupportedExpressionError(
"generation expression cannot contain non utf8 binary collated string
type")
}
+
+ // A write reads the stored text under fixed settings, so that a value
does not depend on the
+ // session that writes it. The text has to read the same there. This also
applies to a table
+ // that computes the values itself, so that the text means the same to any
reader.
+ if (!StoredExpressionText.readsAsInSession(originalSQL)) {
+ throw unsupportedExpressionError(
+ "a write reads it with the default values of the parser configs, such
as " +
+ "spark.sql.parser.escapedStringLiterals and
spark.sql.ansi.enforceReservedKeywords, " +
+ "and this session reads it differently")
+ }
}
// Convert the generation expression to V2 GenerationExpression. The V2
expression is built from
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
new file mode 100644
index 000000000000..cdf189812a40
--- /dev/null
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.{Analyzer,
ExpressionWithUnresolvedIdentifier}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
Expression, ExprId, NamedExpression}
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation,
LogicalPlan, Project}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Reads the SQL text of a generation expression stored with a table,
independently of the
+ * settings of the session that reads it.
+ */
+object StoredExpressionText {
+
+ /**
+ * The settings a stored text is read under. The session time zone, case
sensitivity, timestamp
+ * type and decimal precision loss stay with the given session, because the
settings of the DDL
+ * are not known here and pinning them would silently change existing tables
of a deployment that
+ * sets them. Any other config that the parsing or the analysis reads takes
its default value,
+ * also one that a function reads while it is built, such as
+ * `spark.sql.legacy.negativeIndexInArrayInsert`. With
+ * `spark.sql.legacy.useCurrentConfigsForGeneratedColumn`, these are the
settings of the session.
+ */
+ def conf(session: SQLConf): SQLConf = {
+ if (session.useCurrentSQLConfigsForGeneratedColumn) {
+ session.clone()
+ } else {
+ val c = new SQLConf
+ Analyzer.retainResolutionConfigsForAnalysis(newConf = c, existingConf =
session)
+ c.setConfString(SQLConf.ANSI_ENABLED.key, "true")
+ c.setConfString(SQLConf.SESSION_LOCAL_TIMEZONE.key,
session.sessionLocalTimeZone)
+ c.setConfString(SQLConf.CASE_SENSITIVE.key,
session.caseSensitiveAnalysis.toString)
+ c.setConf(SQLConf.TIMESTAMP_TYPE,
session.getConf(SQLConf.TIMESTAMP_TYPE))
+ c.setConf(
+ SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS,
session.decimalOperationsAllowPrecisionLoss)
+ c
+ }
+ }
+
+ /**
+ * Parses `text` under [[conf]]. A text these settings cannot parse is
parsed under the active
+ * session, as a text they cannot resolve is analyzed under it.
+ */
+ def parse(text: String): Expression = parse(conf(SQLConf.get), text)
+
+ /**
+ * Parses `text` under [[conf]] with the ANSI value of the active session,
for a text that
+ * [[conf]] cannot resolve and the session analyzes. Only its ANSI behavior
follows the session.
+ */
+ def parseWithSessionAnsi(text: String): Expression =
parse(confWithSessionAnsi(SQLConf.get), text)
+
+ private def parse(c: SQLConf, text: String): Expression = {
+ try {
+ SQLConf.withExistingConf(c) {
+ CatalystSqlParser.parseExpression(text)
+ }
+ } catch {
+ case _: ParseException => CatalystSqlParser.parseExpression(text)
+ }
+ }
+
+ private def confWithSessionAnsi(session: SQLConf): SQLConf = {
+ val c = conf(session)
+ c.setConfString(SQLConf.ANSI_ENABLED.key, session.ansiEnabled.toString)
+ c
+ }
+
+ /**
+ * Analyzes `expr` over `input` with `analyzer` under [[conf]], so that an
auto-filled value and
+ * the check of a provided value read a stored text the same way. If these
settings cannot
+ * analyze or resolve it, such as an identifier string that only the session
can parse, or it
+ * analyzes to another plan, such as an aggregate that a catalog stores,
this returns `inSession`
+ * instead, and the analysis of the write resolves that under the session.
+ */
+ def analyze[E <: Expression](
+ expr: E,
+ inSession: => E,
+ input: Seq[Attribute],
+ analyzer: LogicalPlan => LogicalPlan): E = {
+ val named = expr match {
+ case n: NamedExpression => n
+ case e => Alias(e, "value")()
+ }
+ val analyzed = try {
+ SQLConf.withExistingConf(conf(SQLConf.get)) {
+ analyzer(Project(Seq(named), LocalRelation(input)))
+ } match {
+ case Project(Seq(a), _: LocalRelation) if a.resolved => Some(a)
+ case _ => None
+ }
+ } catch {
+ case _: AnalysisException => None
+ }
+ analyzed.map {
+ case Alias(child, _) if !expr.isInstanceOf[NamedExpression] => child
+ case a => a
+ }.getOrElse(inSession).asInstanceOf[E]
+ }
+
+ /**
+ * Whether `text` reads the same under [[conf]] as it does in the active
session. The comparison
+ * uses the ANSI value of the session, so that it is about how the text
reads.
+ */
+ def readsAsInSession(text: String): Boolean = {
+ val fixed = confWithSessionAnsi(SQLConf.get)
+ val parsedInSession = CatalystSqlParser.parseExpression(text)
+ try {
+ val parsed = SQLConf.withExistingConf(fixed) {
+ CatalystSqlParser.parseExpression(text)
+ }
+ normalize(parsed) == normalize(parsedInSession)
+ } catch {
+ case _: ParseException => false
+ }
+ }
+
+ private def normalize(e: Expression): Expression = e.transform {
+ case a: Alias => a.withExprId(ExprId(0))
+ // Two parses of the same text never share the builder, which is a
function.
+ case i: ExpressionWithUnresolvedIdentifier => i.copy(exprBuilder =
noBuilder)
+ }
+
+ private val noBuilder: (Seq[String], Seq[Expression]) => Expression = (_,
exprs) => exprs.head
+}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index 4af1c7d1a13d..bae1bbeea1fa 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -1260,6 +1260,10 @@ private[sql] object QueryCompilationErrors extends
QueryErrorsBase with Compilat
unsupportedTableOperationError(table.name(), "batch scan")
}
+ def unsupportedTableOperationError(table: Table, operation: String):
Throwable = {
+ unsupportedTableOperationError(table.name(), operation)
+ }
+
def unsupportedBatchWriteError(table: Table): Throwable = {
unsupportedTableOperationError(table.name(), "batch write")
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 0ee5338a0e1c..2ad8e618f946 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -4534,6 +4534,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val USE_CURRENT_SQL_CONFIGS_FOR_GENERATED_COLUMN =
+ buildConf("spark.sql.legacy.useCurrentConfigsForGeneratedColumn")
+ .internal()
+ .doc("When true, a write parses and analyzes the SQL text of a
generation expression of a " +
+ "Data Source V2 table under the SQL configs of the current active
SparkSession, and " +
+ "CREATE TABLE and REPLACE TABLE do not reject a text that the session
reads differently.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
+
val STORE_ANALYZED_PLAN_FOR_VIEW =
buildConf("spark.sql.legacy.storeAnalyzedPlanForView")
.internal()
@@ -10029,6 +10040,9 @@ class SQLConf extends Serializable with Logging with
SqlApiConf {
def useCurrentSQLConfigsForView: Boolean =
getConf(SQLConf.USE_CURRENT_SQL_CONFIGS_FOR_VIEW)
+ def useCurrentSQLConfigsForGeneratedColumn: Boolean =
+ getConf(SQLConf.USE_CURRENT_SQL_CONFIGS_FOR_GENERATED_COLUMN)
+
def storeAnalyzedPlanForView: Boolean =
getConf(SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW)
def allowAutoGeneratedAliasForView: Boolean =
getConf(SQLConf.ALLOW_AUTO_GENERATED_ALIAS_FOR_VEW)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
index de11b78452cc..1026d0aa2648 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
@@ -26,7 +26,7 @@ import org.apache.spark.sql.types.{TimestampNTZType,
TimestampType}
class GeneratedColumnExpressionSuite extends SparkFunSuite {
private def genExpr(childType: DataType): GeneratedColumnExpression =
- GeneratedColumnExpression(Literal.create(null, childType), "<gen expr>")
+ GeneratedColumnExpression(Literal.create(null, childType), "NULL")
test("SPARK-57303: validate accepts a lossless widening to a nanosecond
timestamp column") {
// The generation expression's type is up-castable to the column type, so
validate() succeeds:
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
index 1fd1f84c48e1..dbbf210ab190 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
@@ -1551,7 +1551,7 @@ class DataSourceV2DataFrameSuite
// Unlike the disabled case above, this behavior is not parameterized over
ANSI: reserved keyword
// enforcement (enforceReservedKeywords = ansiEnabled &&
ENFORCE_RESERVED_KEYWORDS) can only be on
// when ANSI is enabled. With ANSI disabled the conf is a no-op.
- test("generation expression resolves a reserved keyword to the function when
" +
+ test("generation expression with a reserved keyword is rejected when " +
"enforceReservedKeywords is enabled") {
val tableName = "testcat.ns1.ns2.tbl"
withSQLConf(
@@ -1559,21 +1559,26 @@ class DataSourceV2DataFrameSuite
SQLConf.ENFORCE_RESERVED_KEYWORDS.key -> "true") {
withTable(tableName) {
// With reserved keyword enforcement on, the unquoted `current_date`
in the generation
- // expression binds to the built-in CURRENT_DATE function rather than
the same-named
- // column. This matches the SQL standard.
- val createExec = executeAndKeepPhysicalPlan[CreateTableExec] {
- sql(
- s"""CREATE TABLE $tableName (
- | `current_date` DATE,
- | c2 DATE GENERATED ALWAYS AS (current_date)
- |) USING foo""".stripMargin)
- }
- val genExpr = createExec.columns.find(_.name ==
"c2").get.columnGenerationExpression()
- assert(genExpr != null)
- assert(genExpr.getSql == "current_date")
- // CURRENT_DATE has no V2 representation, so the connector falls back
to the SQL form. A
- // non-null field reference here would mean the column incorrectly
took priority.
- assert(genExpr.getExpression == null)
+ // expression binds to the built-in CURRENT_DATE function. A write
reads the stored text
+ // without the enforcement and would bind it to the same-named column
instead.
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(
+ s"""CREATE TABLE $tableName (
+ | `current_date` DATE,
+ | c2 DATE GENERATED ALWAYS AS (current_date)
+ |) USING foo""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "c2",
+ "expressionStr" -> "current_date",
+ "reason" -> ("a write reads it with the default values of the
parser configs, " +
+ "such as spark.sql.parser.escapedStringLiterals and " +
+ "spark.sql.ansi.enforceReservedKeywords, and this session reads
it differently")))
+
+ // CURDATE is not a reserved keyword, so it reads as the function in
both cases.
+ sql(s"CREATE TABLE $tableName (c2 DATE GENERATED ALWAYS AS
(curdate())) USING foo")
}
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index accba28d42da..525b02bfb98f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -17,15 +17,19 @@
package org.apache.spark.sql.connector
-import org.apache.spark.SparkRuntimeException
+import java.util.Collections
+
+import org.apache.spark.{SparkRuntimeException, SparkThrowable}
import org.apache.spark.sql.{AnalysisException, DataFrame, QueryTest, Row}
import org.apache.spark.sql.catalyst.QueryPlanningTracker
import org.apache.spark.sql.catalyst.expressions.CheckInvariant
import org.apache.spark.sql.catalyst.util.GeneratedColumn
-import org.apache.spark.sql.connector.catalog.{Identifier,
InMemoryRowLevelOperationTableCatalog}
+import org.apache.spark.sql.connector.catalog.{Column, GenerationExpression,
Identifier, InMemoryCatalog, InMemoryRowLevelOperationTableCatalog}
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
+import org.apache.spark.sql.connector.expressions.{Expression => V2Expression,
GeneralScalarExpression, Transform}
import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, MetadataBuilder, StringType,
StructType}
/**
* Tests for generated column auto-fill and constraint enforcement during
writes.
@@ -110,6 +114,15 @@ class GeneratedColumnWriteSuite extends QueryTest with
DatasourceV2SQLBase {
s"generation expression leaked into the schema of $column:
${metadata.json}")
}
+ private def assertWriteFails(condition: String)(write: => Unit): Unit = {
+ val ex = intercept[SparkThrowable](write)
+ assert(ex.getCondition == condition)
+ }
+
+ private val readsDifferentlyReason = "a write reads it with the default
values of the parser " +
+ "configs, such as spark.sql.parser.escapedStringLiterals and " +
+ "spark.sql.ansi.enforceReservedKeywords, and this session reads it
differently"
+
testGeneratedColumnWrite("append_data_v2") { table =>
import testImplicits._
Seq(Tuple5(1L, "foo", "2020-10-11 12:30:30", 100, "2020-11-12"))
@@ -1351,4 +1364,559 @@ class GeneratedColumnWriteSuite extends QueryTest with
DatasourceV2SQLBase {
"No CheckInvariant should be added when the table lacks the write
capability")
}
}
+
+ test("SPARK-60757: an auto-filled value does not depend on the parser
settings of the writer") {
+ import testImplicits._
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (concat(a, 'x\\\\y'))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('default')")
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('insert')")
+ Seq("append").toDF("a").writeTo(s"testcat.$tblName").append()
+ }
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Seq("default", "insert", "append").map(a => Row(a, a + "x\\y")))
+
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"INSERT OVERWRITE testcat.$tblName(a) VALUES ('overwrite')")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("overwrite",
"overwritex\\y"))
+ }
+ }
+
+ test("SPARK-60757: the validation of a provided value does not depend on the
parser settings " +
+ "of the writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (concat(a, 'x\\\\y'))
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"INSERT INTO testcat.$tblName(a, b) SELECT 'p', concat('p', 'x',
chr(92), 'y')")
+ assertWriteFails("CHECK_CONSTRAINT_VIOLATION") {
+ sql(s"INSERT INTO testcat.$tblName(a, b) " +
+ "SELECT 'q', concat('q', 'x', chr(92), chr(92), 'y')")
+ }
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("p", "px\\y"))
+ }
+ }
+
+ test("SPARK-60757: a later write carries a row that a non-ANSI writer
auto-filled") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b DOUBLE GENERATED ALWAYS AS (a + 1)
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('9007199254740993')")
+ sql(s"INSERT INTO testcat.$tblName SELECT * FROM testcat.$tblName")
+ }
+ val row = Row("9007199254740993", 9007199254740994D)
+ checkAnswer(spark.table(s"testcat.$tblName"), Seq(row, row))
+ }
+ }
+
+ test("SPARK-60757: a DELETE does not check the rows that it carries") {
+ withRowLevelCatalog {
+ val tblName = "my_tab"
+ withTable(s"$rowLevelCat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ // Without the capability, the rows go in as a non-ANSI writer
stored them before.
+ sql(s"""CREATE TABLE $rowLevelCat.$tblName(
+ | id INT,
+ | a STRING,
+ | b DOUBLE GENERATED ALWAYS AS (a + 1)
+ |) USING foo TBLPROPERTIES ('$noWriteCapProp' =
'false')""".stripMargin)
+ sql(s"INSERT INTO $rowLevelCat.$tblName VALUES " +
+ "(1, 'abc', NULL), (2, '2', 3D), (3, '3', 4D)")
+ sql(s"ALTER TABLE $rowLevelCat.$tblName SET TBLPROPERTIES
('$noWriteCapProp' = 'true')")
+ sql(s"DELETE FROM $rowLevelCat.$tblName WHERE id >= 3")
+ }
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ sql(s"DELETE FROM $rowLevelCat.$tblName WHERE id >= 2")
+ }
+ checkAnswer(spark.table(s"$rowLevelCat.$tblName"), Row(1, "abc", null))
+ assertWriteFails("CAST_INVALID_INPUT") {
+ sql(s"INSERT INTO $rowLevelCat.$tblName SELECT * FROM
$rowLevelCat.$tblName")
+ }
+ }
+ }
+ }
+
+ test("SPARK-60757: an explicit cast and arithmetic keep failing under a
non-ANSI writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b INT GENERATED ALWAYS AS (CAST(a AS INT)),
+ | c INT,
+ | d INT GENERATED ALWAYS AS (c + 1)
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ assertWriteFails("CAST_INVALID_INPUT") {
+ sql(s"INSERT INTO testcat.$tblName(a, c) VALUES ('abc', 1)")
+ }
+ assertWriteFails("ARITHMETIC_OVERFLOW") {
+ sql(s"INSERT INTO testcat.$tblName(a, c) VALUES ('1',
${Int.MaxValue})")
+ }
+ sql(s"INSERT INTO testcat.$tblName(a, c) VALUES ('1', 1)")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("1", 1, 1, 2))
+ }
+ }
+
+ test("SPARK-60757: the validation of a provided value keeps failing under a
non-ANSI writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ // UPPER has no way back from its V2 form, so the validation reads the
SQL text.
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b INT GENERATED ALWAYS AS (CAST(upper(a) AS INT))
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ assertWriteFails("CAST_INVALID_INPUT") {
+ sql(s"INSERT INTO testcat.$tblName(a, b) VALUES ('abc', NULL)")
+ }
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Nil)
+ }
+ }
+
+ test("SPARK-60757: a reserved keyword of an auto-filled column does not
follow the settings " +
+ "of the writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | `current_date` DATE,
+ | b DATE GENERATED ALWAYS AS (current_date)
+ |) USING foo""".stripMargin)
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.ENFORCE_RESERVED_KEYWORDS.key -> "true") {
+ sql(s"INSERT INTO testcat.$tblName(`current_date`) VALUES
(DATE'2024-06-15')")
+ }
+ val date = sqlDate("2024-06-15")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(date, date))
+ }
+ }
+
+ test("SPARK-60757: an implicit cast of an auto-filled column does not follow
a non-ANSI writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b BIGINT GENERATED ALWAYS AS (a + 1)
+ |) USING foo""".stripMargin)
+ }
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ assertWriteFails("CAST_INVALID_INPUT") {
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('abc')")
+ }
+ // A non-ANSI cast to DOUBLE would lose the last digit.
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('9007199254740993')")
+ }
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Row("9007199254740993", 9007199254740994L))
+ }
+ }
+
+ test("SPARK-60757: a function of an auto-filled column does not follow the
settings of the " +
+ "writer") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a INT,
+ | arr ARRAY<INT>,
+ | b INT GENERATED ALWAYS AS (abs(a)),
+ | c INT GENERATED ALWAYS AS (element_at(arr, 2)),
+ | d INT GENERATED ALWAYS AS (size(arr))
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ assertWriteFails("ARITHMETIC_OVERFLOW") {
+ sql(s"INSERT INTO testcat.$tblName(a, arr) VALUES (${Int.MinValue},
array(1, 2))")
+ }
+ assertWriteFails("INVALID_ARRAY_INDEX_IN_ELEMENT_AT") {
+ sql(s"INSERT INTO testcat.$tblName(a, arr) VALUES (1, array(1))")
+ }
+ }
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "false",
+ SQLConf.LEGACY_SIZE_OF_NULL.key -> "true") {
+ sql(s"INSERT INTO testcat.$tblName(a, arr) VALUES (1, NULL)")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(1, null, 1, null,
null))
+ }
+ }
+
+ test("SPARK-60757: a temporary function does not replace a function of an
auto-filled column") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (upper(a))
+ |) USING foo""".stripMargin)
+ spark.udf.register("upper", (s: String) => s + "-temp")
+ try {
+ withSQLConf(SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "first") {
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('x')")
+ }
+ } finally {
+ sql("DROP TEMPORARY FUNCTION IF EXISTS upper")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("x", "X"))
+ }
+ }
+
+ test("SPARK-60757: an IDENTIFIER clause is not a different reading of a
generation expression") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a INT,
+ | b INT GENERATED ALWAYS AS (IDENTIFIER('a') + 1)
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES (1)")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(1, 2))
+ }
+ }
+
+ test("SPARK-60757: CREATE and REPLACE TABLE reject a generation expression
the session reads " +
+ "differently") {
+ val tblName = "my_tab"
+ for {
+ (confs, genExpr) <- Seq(
+ (Seq(SQLConf.ESCAPED_STRING_LITERALS.key -> "true"), "concat(a,
'x\\\\y')"),
+ (Seq(SQLConf.ANSI_ENABLED.key -> "true",
SQLConf.DOUBLE_QUOTED_IDENTIFIERS.key -> "true"),
+ "concat(\"a\", 'x')"))
+ create <- Seq("CREATE TABLE", "CREATE OR REPLACE TABLE")
+ } withSQLConf(confs: _*) {
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"""$create testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS ($genExpr)
+ |) USING foo""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "b",
+ "expressionStr" -> genExpr,
+ "reason" -> readsDifferentlyReason))
+ }
+ }
+
+ test("SPARK-60757: createTable with a schema rejects a generation expression
the session " +
+ "reads differently") {
+ val metadata = new MetadataBuilder()
+ .putString(GeneratedColumn.GENERATION_EXPRESSION_METADATA_KEY,
"concat(a, 'x\\\\y')")
+ .build()
+ val schema = new StructType()
+ .add("a", StringType)
+ .add("b", StringType, nullable = true, metadata)
+ val tblName = "my_tab"
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ spark.catalog.createTable(s"testcat.$tblName", "foo", schema,
Map.empty[String, String])
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "b",
+ "expressionStr" -> "concat(a, 'x\\\\y')",
+ "reason" -> readsDifferentlyReason))
+ }
+ }
+
+ test("SPARK-60757: a stored text that is not a scalar expression fails the
analysis of the " +
+ "write") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ // A catalog can store a text that CREATE TABLE rejects.
+ catalog("testcat").asInstanceOf[InMemoryCatalog].createTable(
+ Identifier.of(Array(), tblName),
+ Array(
+ Column.create("i", IntegerType),
+ Column.create("b", IntegerType, true, null, "sum(i)", null)),
+ Array.empty[Transform],
+ Collections.emptyMap[String, String])
+ assertWriteFails("MISSING_GROUP_BY") {
+ sql(s"INSERT INTO testcat.$tblName(i) VALUES (1)")
+ }
+ }
+ }
+
+ test("SPARK-60757: a parser setting of the session does not reject an
expression that reads " +
+ "the same") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (concat(a, 'xy'))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('v')")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("v", "vxy"))
+ }
+ }
+
+ test("SPARK-60757: the validation of a provided value reads the text of a
table with a V2 form") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ // The V2 form is COALESCE(s, '1'), with the implicit cast of a
non-ANSI session.
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | s STRING,
+ | b STRING GENERATED ALWAYS AS (coalesce(s, 1))
+ |) USING foo""".stripMargin)
+ val table =
catalog("testcat").asTableCatalog.loadTable(Identifier.of(Array(), tblName))
+ val genExpr = table.columns().find(_.name ==
"b").get.columnGenerationExpression()
+ assert(genExpr.getExpression != null)
+ sql(s"INSERT INTO testcat.$tblName VALUES ('007', '7')")
+ sql(s"INSERT INTO testcat.$tblName(s) VALUES ('008')")
+ checkAnswer(spark.table(s"testcat.$tblName"), Seq(Row("007", "7"),
Row("008", "8")))
+ }
+ }
+ }
+
+ test("SPARK-60757: the validation of a provided value does not substitute a
variable") {
+ import testImplicits._
+ val tblName = "my_tab"
+ val variable = s"$${system:java.version}"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.VARIABLE_SUBSTITUTE_ENABLED.key -> "false") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (concat(a, '$variable'))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName VALUES ('w', 'w$variable')")
+ }
+ // Substitution is on by default. A DataFrame write keeps the variable
in its values.
+ Seq(("v", s"v$variable")).toDF("a",
"b").writeTo(s"testcat.$tblName").append()
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Seq(Row("w", s"w$variable"), Row("v", s"v$variable")))
+ }
+ }
+
+ test("SPARK-60757: a text the fixed settings cannot resolve is read under
the session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | d DATE,
+ | s STRING,
+ | b BOOLEAN GENERATED ALWAYS AS (CAST(d AS BOOLEAN)),
+ | c INT GENERATED ALWAYS AS (CAST(s AS INT))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(d, s) VALUES (DATE'2024-01-01',
'1')")
+ sql(s"INSERT INTO testcat.$tblName VALUES (DATE'2024-01-02', '2',
NULL, 2)")
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Seq(
+ Row(sqlDate("2024-01-01"), "1", null, 1),
+ Row(sqlDate("2024-01-02"), "2", null, 2)))
+ // The other column of the same write stays with the fixed settings.
+ assertWriteFails("CAST_INVALID_INPUT") {
+ sql(s"INSERT INTO testcat.$tblName(d, s) VALUES (DATE'2024-01-03',
'abc')")
+ }
+ }
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ assertWriteFails("DATATYPE_MISMATCH.CAST_WITH_CONF_SUGGESTION") {
+ sql(s"INSERT INTO testcat.$tblName(d, s) VALUES (DATE'2024-01-04',
'4')")
+ }
+ }
+ }
+ }
+
+ test("SPARK-60757: a text the fixed settings cannot parse is read under the
session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ // A table created before the DDL rejected this text.
+ catalog("testcat").asInstanceOf[InMemoryCatalog].createTable(
+ Identifier.of(Array(), tblName),
+ Array(
+ Column.create("s", new StructType().add("f", IntegerType)),
+ Column.create("b", IntegerType, true, null, "s.\"f\"", null)),
+ Array.empty[Transform],
+ Collections.emptyMap[String, String])
+ val autoFilled = s"INSERT INTO testcat.$tblName(s) VALUES
(named_struct('f', 1))"
+ val provided = s"INSERT INTO testcat.$tblName VALUES (named_struct('f',
1), 1)"
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.DOUBLE_QUOTED_IDENTIFIERS.key -> "true") {
+ assert(!hasCheckInvariant(autoFilled))
+ assert(hasCheckInvariant(provided))
+ }
+ withSQLConf(SQLConf.DOUBLE_QUOTED_IDENTIFIERS.key -> "false") {
+ assertWriteFails("PARSE_SYNTAX_ERROR")(hasCheckInvariant(autoFilled))
+ }
+ }
+ }
+
+ test("SPARK-60757: a text the fixed settings cannot resolve keeps their
parser settings") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "false") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | d DATE,
+ | s STRING,
+ | b BOOLEAN GENERATED ALWAYS AS (CAST(d AS BOOLEAN) OR s =
'x\\\\y')
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"INSERT INTO testcat.$tblName(d, s) " +
+ "SELECT DATE'2024-01-01', concat('x', chr(92), 'y')")
+ sql(s"INSERT INTO testcat.$tblName " +
+ "SELECT DATE'2024-01-02', concat('x', chr(92), 'y'), true")
+ }
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Seq(
+ Row(sqlDate("2024-01-01"), "x\\y", true),
+ Row(sqlDate("2024-01-02"), "x\\y", true)))
+ }
+ }
+ }
+
+ test("SPARK-60757: a text the fixed settings cannot analyze is read under
the session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.DOUBLE_QUOTED_IDENTIFIERS.key -> "true") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a INT,
+ | b INT GENERATED ALWAYS AS (IDENTIFIER('"a"') + 1)
+ |) USING foo""".stripMargin)
+ assert(!hasCheckInvariant(s"INSERT INTO testcat.$tblName(a) VALUES
(1)"))
+ assert(hasCheckInvariant(s"INSERT INTO testcat.$tblName VALUES (5,
6)"))
+ }
+ }
+ }
+
+ test("SPARK-60757: a V2 expression without a text that does not convert
names its column") {
+ val tblName = "my_tab"
+ val genExpr = new GeneralScalarExpression("NOT_A_FUNCTION",
Array.empty[V2Expression])
+ withTable(s"testcat.$tblName") {
+ catalog("testcat").asInstanceOf[InMemoryCatalog].createTable(
+ Identifier.of(Array(), tblName),
+ Array(
+ Column.create("a", IntegerType),
+ Column.builderFor("b", IntegerType)
+ .generationExpression(new GenerationExpression(genExpr)).build()),
+ Array.empty[Transform],
+ Collections.emptyMap[String, String])
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"INSERT INTO testcat.$tblName VALUES (1, 2)")
+ },
+ condition = "UNSUPPORTED_FEATURE.TABLE_OPERATION",
+ parameters = Map(
+ "tableName" -> "`testcat`.`my_tab`",
+ "operation" -> ("a value for the generated column `b`, because Spark
cannot convert " +
+ s"its expression $genExpr")))
+ }
+ }
+
+ test("SPARK-60757: the validation of a provided value compares it with the
stored value") {
+ val tblName = "my_tab"
+ Seq("false", "true").foreach { ansi =>
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | s STRING,
+ | b STRING GENERATED ALWAYS AS (coalesce(s, 1))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName VALUES ('007', '7')")
+ assertWriteFails("CHECK_CONSTRAINT_VIOLATION") {
+ sql(s"INSERT INTO testcat.$tblName VALUES ('007', '07')")
+ }
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("007", "7"))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60757: the check of a provided value reports the error of the
expression") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a INT,
+ | c DECIMAL(20, 0) GENERATED ALWAYS AS (a div 0)
+ |) USING foo""".stripMargin)
+ assertWriteFails("DIVIDE_BY_ZERO") {
+ sql(s"INSERT INTO testcat.$tblName VALUES (1, 0)")
+ }
+ }
+ }
+
+ test("SPARK-60757: a legacy config reads a generation expression under the
session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(
+ SQLConf.USE_CURRENT_SQL_CONFIGS_FOR_GENERATED_COLUMN.key -> "true",
+ SQLConf.ESCAPED_STRING_LITERALS.key -> "true") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b STRING GENERATED ALWAYS AS (concat(a, 'x\\\\y'))
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('v')")
+ // The check of a provided value reads the text under the session too.
+ sql(s"INSERT INTO testcat.$tblName VALUES ('w', 'wx\\\\y')")
+ }
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Seq(Row("v", "vx\\\\y"), Row("w", "wx\\\\y")))
+ }
+ }
+
+ test("SPARK-60757: a generation expression reads the timestamp type of the
session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "America/Los_Angeles",
+ SQLConf.TIMESTAMP_TYPE.key ->
SQLConf.TimestampTypes.TIMESTAMP_NTZ.toString) {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a STRING,
+ | b TIMESTAMP_NTZ GENERATED ALWAYS AS (to_timestamp(a))
+ |) USING foo""".stripMargin)
+ // This time does not exist in the time zone, so a TIMESTAMP_LTZ value
would read 03:30.
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES ('2024-03-10 02:30:00')")
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Row("2024-03-10 02:30:00",
java.time.LocalDateTime.parse("2024-03-10T02:30:00")))
+ assertWriteFails("CHECK_CONSTRAINT_VIOLATION") {
+ sql(s"INSERT INTO testcat.$tblName VALUES " +
+ "('2024-03-10 02:30:00', TIMESTAMP_NTZ'2024-03-10 03:30:00')")
+ }
+ }
+ }
+ }
+
+ test("SPARK-60757: a generation expression reads the decimal precision loss
of the session") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ withSQLConf(SQLConf.DECIMAL_OPERATIONS_ALLOW_PREC_LOSS.key -> "false") {
+ // Without the precision loss a * a is DECIMAL(38, 20), and with it
DECIMAL(38, 17).
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a DECIMAL(20, 10),
+ | b DECIMAL(38, 20) GENERATED ALWAYS AS (a * a)
+ |) USING foo""".stripMargin)
+ sql(s"INSERT INTO testcat.$tblName(a) VALUES (1.1234567891)")
+ checkAnswer(
+ spark.table(s"testcat.$tblName"),
+ Row(new java.math.BigDecimal("1.1234567891"),
+ new java.math.BigDecimal("1.26215515697488187881")))
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]