This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 019733144b0a [SPARK-58460][SQL] Do not leak generated column 
expressions into DataFrame schemas and derived tables
019733144b0a is described below

commit 019733144b0a8f18ff333fe9d5076f16dd3d3311
Author: Szehon Ho <[email protected]>
AuthorDate: Fri Jul 31 19:11:14 2026 +0800

    [SPARK-58460][SQL] Do not leak generated column expressions into DataFrame 
schemas and derived tables
    
    ### What changes were proposed in this pull request?
    
    A generated column's generation expression was carried as 
`GENERATION_EXPRESSION` metadata on a relation's output attributes. This PR 
treats that key as internal metadata, so it never reaches a relation's output, 
and reworks the write path to take the expression from the table's V2 columns 
instead:
    
    - `GENERATION_EXPRESSION` is added to `INTERNAL_METADATA_KEYS`, so 
`removeInternalMetadata` strips it. `DataSourceV2Relation.create` already ran 
the table schema through that, so batch relations are clean; the 
`StreamingRelationV2` paths built attributes directly from 
`table.columns.toAttributes` and now use a new `toOutputAttributes`, which 
applies the same strip (keeping field IDs, which the column-ID feature 
deliberately exposes).
    - `ResolveOutputRelation` attaches the expressions to the expected output 
from `table.columns()` (the persisted source of truth) via 
`GeneratedColumn.attachGenerationExpressions`, only for as long as resolving 
the write takes, so `TableOutputResolver` can still auto-fill missing generated 
columns.
    - Instead of removing metadata from auto-filled columns to suppress 
validation, the columns Spark computed are now marked with a new internal key, 
`__auto_filled_generated_column`, and `ResolveTableConstraints` skips those. 
Validation is therefore the default: any value that did not come from the 
generation expression gets a `CheckInvariant`, as before.
    
    ### Why are the changes needed?
    
    Because the expression rode along on the relation's output, it leaked in 
two user-visible ways.
    
    It showed up in a DataFrame's schema:
    
    ```scala
    sql("CREATE TABLE testcat.t (id INT, doubled INT GENERATED ALWAYS AS (id * 
2)) USING foo")
    spark.table("testcat.t").schema("doubled").metadata
    // {"GENERATION_EXPRESSION":"id * 2"}
    ```
    
    And, since a table created from a query derives its columns from that 
schema, the generated column was promoted back into a real generated column on 
the new table:
    
    ```scala
    sql("CREATE TABLE testcat.dst USING foo AS SELECT * FROM testcat.t")
    // dst.doubled is a generated column copied from the source
    
    spark.readStream.table("testcat.t").writeStream.toTable("testcat.dst2")
    // fails: the target is created with a generated column, and streaming 
writes to
    // tables with generated columns are unsupported
    ```
    
    Generated column values on writes were added recently (SPARK-57644) and are 
not in any release, so this corrects the behavior of an unreleased feature.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, within unreleased branches only. `GENERATION_EXPRESSION` no longer 
appears in the metadata of a DataFrame's schema (batch reads, `readStream`, and 
`SELECT`), and `CREATE TABLE AS SELECT` / `writeStream.toTable` now create 
ordinary columns rather than inheriting the source's generated columns. 
Auto-filling and validation of generated columns on writes to an existing table 
are unchanged.
    
    ### How was this patch tested?
    
    `GeneratedColumnWriteSuite` (72 tests, all passing), with three new cases:
    
    - generation expression is not exposed in the read schema, covering 
`spark.table`, `spark.read.table`, `SELECT`, and `readStream.table`
    - CTAS from a table with generated columns does not create generated 
columns, and the resulting column behaves as an ordinary one
    - streaming write to a new table does not create generated columns
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Cursor 2.5, Opus 5
    
    Closes #57662 from szehon-ho/gencol-metadata-not-propagated.
    
    Authored-by: Szehon Ho <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
    (cherry picked from commit 9586a2a0c2b170e4caa2aa70b3bb791ca99330f5)
    Signed-off-by: Wenchen Fan <[email protected]>
---
 .../spark/sql/catalyst/analysis/Analyzer.scala     | 26 ++-----
 .../sql/catalyst/analysis/RelationResolution.scala |  4 +-
 .../analysis/ResolveTableConstraints.scala         | 22 +++---
 .../spark/sql/catalyst/util/GeneratedColumn.scala  | 88 +++++++++++++++++++---
 .../apache/spark/sql/catalyst/util/package.scala   |  4 +-
 .../sql/connector/catalog/CatalogV2Implicits.scala | 11 ++-
 .../sql/catalyst/analysis/ResolveDataSource.scala  |  5 +-
 .../sql/connector/GeneratedColumnWriteSuite.scala  | 79 ++++++++++++++++++-
 8 files changed, 186 insertions(+), 53 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index ec17902e2e85..0369a9ce1a5f 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -3935,12 +3935,11 @@ class Analyzer(
         val defaultValueFillMode =
           if (conf.coerceInsertNestedTypes && v2Write.schemaEvolutionEnabled) 
RECURSE
           else FILL
-        // Only let TableOutputResolver see generation expression metadata if 
the table
-        // supports auto-filling generated columns on write.
+        // Generation expressions live on the table's columns, so attach them 
to the expected
+        // output for TableOutputResolver, which auto-fills the generated 
columns the query is
+        // missing.
         val expected = v2Write.table match {
-          case r: DataSourceV2Relation
-            if !GeneratedColumn.supportsGeneratedColumnsOnWrite(r.table) =>
-            r.output.map(GeneratedColumn.removeGenerationExpressionMetadata)
+          case r: DataSourceV2Relation => 
GeneratedColumn.attachGenerationExpressions(r)
           case _ => v2Write.table.output
         }
         val (projection, autoFilledGenCols) =
@@ -3950,20 +3949,9 @@ class Analyzer(
         if (projection != v2Write.query) {
           val cleanedTable = v2Write.table match {
             case r: DataSourceV2Relation =>
-              r.copy(output = r.output.map { attr =>
-                val cleaned = CharVarcharUtils.cleanAttrMetadata(attr)
-                // Strip the generation expression metadata from columns Spark 
auto-filled, so
-                // ResolveTableConstraints does not add a (redundant) 
CheckInvariant for them:
-                // their values were computed from the generation expression 
and are correct by
-                // construction. User-provided generated columns keep the 
metadata so their
-                // values are still validated.
-                if (autoFilledGenCols.contains(attr.name)) {
-                  GeneratedColumn.removeGenerationExpressionMetadata(cleaned)
-                    .asInstanceOf[AttributeReference]
-                } else {
-                  cleaned
-                }
-              })
+              val cleaned = r.output.map(CharVarcharUtils.cleanAttrMetadata)
+              r.copy(output =
+                GeneratedColumn.markAutoFilledGeneratedColumns(cleaned, 
autoFilledGenCols))
             case other => other
           }
           v2Write.withNewQuery(projection).withNewTable(cleanedTable)
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
index 0a085fcc2971..ef6648486310 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
@@ -354,7 +354,7 @@ class RelationResolution(
         val relation = if (u.isStreaming) {
           StreamingRelationV2(
             None, changelogTable.name, changelogTable, u.options,
-            changelogTable.columns.toAttributes, Some(catalog), Some(ident), 
None)
+            changelogTable.columns.toOutputAttributes, Some(catalog), 
Some(ident), None)
         } else {
           DataSourceV2Relation.create(changelogTable, Some(catalog), 
Some(ident), u.options)
         }
@@ -454,7 +454,7 @@ class RelationResolution(
               table.name,
               table,
               options,
-              table.columns.toAttributes,
+              table.columns.toOutputAttributes,
               Some(catalog),
               Some(ident),
               v1Fallback
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
index 2a4b5693c705..580af46533f3 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
@@ -80,10 +80,10 @@ class ResolveTableConstraints(val catalogManager: 
CatalogManager) extends Rule[L
   }
 
   /**
-   * For each user-provided generated column, add a CheckInvariant that 
validates the column
-   * value matches the generation expression. Auto-filled generated columns 
are excluded:
-   * ResolveOutputRelation strips their GENERATION_EXPRESSION metadata from 
the table output,
-   * so they won't appear here.
+   * For each generated column whose value was provided by the write, add a 
CheckInvariant that
+   * validates the column value matches the generation expression. Generated 
columns Spark
+   * auto-filled are excluded: ResolveOutputRelation marks them in the table 
output and their
+   * values are correct by construction.
    */
   private def buildGeneratedColumnConstraints(
       r: DataSourceV2Relation,
@@ -92,20 +92,16 @@ class ResolveTableConstraints(val catalogManager: 
CatalogManager) extends Rule[L
       return Seq.empty
     }
 
-    // Use V2 columns from the table to access both V2 expressions and SQL 
strings.
-    // Only add constraints for generated columns whose GENERATION_EXPRESSION 
metadata
-    // is still present in the table output -- ResolveOutputRelation strips 
the metadata
-    // from auto-filled columns so they are excluded here.
-    val v2Columns = r.table.columns()
     val resolver = catalogManager.v1SessionCatalog.conf.resolver
-    val userProvidedGenCols = r.output
-      .filter(attr => GeneratedColumn.isGeneratedColumn(attr.metadata))
+    val autoFilledGenCols = r.output
+      .filter(GeneratedColumn.isAutoFilledGeneratedColumn)
       .map(_.name)
       .toSet
 
-    v2Columns.flatMap { col =>
+    // Use V2 columns from the table to access both V2 expressions and SQL 
strings.
+    r.table.columns().flatMap { col =>
       Option(col.columnGenerationExpression())
-        .filter(_ => userProvidedGenCols.exists(n => resolver(n, col.name)))
+        .filter(_ => !autoFilledGenCols.exists(n => resolver(n, col.name)))
         .map { genExpr =>
           val catalystExpr = buildGenerationCatalystExpression(genExpr)
           val colRef = UnresolvedAttribute.quoted(col.name)
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala
index f8ca60c3a40a..574d4edb21bb 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GeneratedColumn.scala
@@ -17,11 +17,12 @@
 
 package org.apache.spark.sql.catalyst.util
 
-import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
 import org.apache.spark.sql.catalyst.plans.logical.ColumnDefinition
 import org.apache.spark.sql.connector.catalog.{Column, Identifier, Table, 
TableCapability,
   TableCatalog, TableCatalogCapability}
 import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
 import org.apache.spark.sql.types.{Metadata, MetadataBuilder, StructField, 
StructType}
 
 /**
@@ -35,6 +36,13 @@ object GeneratedColumn {
    */
   val GENERATION_EXPRESSION_METADATA_KEY = "GENERATION_EXPRESSION"
 
+  /**
+   * The metadata key marking a generated column in a write target's output as 
one whose value
+   * Spark computed from the generation expression, so that it is not 
validated against the
+   * expression that produced it. Only set while resolving a write.
+   */
+  val AUTO_FILLED_GENERATED_COLUMN_METADATA_KEY = 
"__auto_filled_generated_column"
+
   /**
    * Whether the given `field` is a generated column
    */
@@ -114,18 +122,74 @@ object GeneratedColumn {
   }
 
   /**
-   * Returns an attribute with the generation expression metadata removed.
-   * Used when the catalog does not support auto-filling generated columns on 
write.
+   * Returns `relation`'s output with every generated column's generation 
expression recorded in
+   * the attribute's metadata, which is where [[TableOutputResolver]] looks 
when it auto-fills the
+   * generated columns a write does not provide a value for.
+   *
+   * Generation expressions are internal metadata: they should neither surface 
in a DataFrame's
+   * schema nor propagate into tables created from it. A table's V2 columns 
are the persisted form
+   * and the source of truth, so the write path copies the expression into 
plan attribute metadata
+   * only for as long as resolving the write takes.
+   *
+   * The output is returned unchanged if the table does not ask Spark to 
handle generated columns.
    */
-  def removeGenerationExpressionMetadata(attr: Attribute): Attribute = {
-    if (isGeneratedColumn(attr.metadata)) {
-      val cleaned = new MetadataBuilder()
-        .withMetadata(attr.metadata)
-        .remove(GENERATION_EXPRESSION_METADATA_KEY)
-        .build()
-      attr.withMetadata(cleaned)
-    } else {
-      attr
+  def attachGenerationExpressions(relation: DataSourceV2Relation): 
Seq[AttributeReference] = {
+    if (!supportsGeneratedColumnsOnWrite(relation.table)) {
+      return relation.output
+    }
+    val genExprs = relation.table.columns()
+      .flatMap(col => Option(col.generationExpression()).map(col.name -> _))
+      .toMap
+    relation.output.map { attr =>
+      genExprs.get(attr.name) match {
+        case Some(genExpr) =>
+          withMetadataEntry(attr, GENERATION_EXPRESSION_METADATA_KEY, genExpr)
+        case None => attr
+      }
+    }
+  }
+
+  /**
+   * When a write supplies its own value for a generated column, that value 
must agree with the
+   * generation expression, so ResolveTableConstraints validates it with a 
CheckInvariant, the same
+   * way it enforces a table's CHECK constraints. In contrast, values computed 
by Spark from the
+   * generation expression pass that check by construction, so this mark is 
what tells the rule to
+   * skip them.
+   *
+   * Returns a write target's `output` with the generated columns named in 
`autoFilled` marked as
+   * computed by Spark, so that it can skip CheckInvariant validation.
+   *
+   * Columns are left unmarked by default, so a value that did not come from 
the generation
+   * expression is always validated.
+   */
+  def markAutoFilledGeneratedColumns(
+      output: Seq[AttributeReference],
+      autoFilled: Set[String]): Seq[AttributeReference] = {
+    output.map { attr =>
+      if (autoFilled.contains(attr.name)) {
+        withMetadataEntry(attr, AUTO_FILLED_GENERATED_COLUMN_METADATA_KEY, 
"true")
+      } else {
+        attr
+      }
     }
   }
+
+  /**
+   * Whether `attr` is a generated column whose value Spark computed from the 
generation
+   * expression (see [[markAutoFilledGeneratedColumns]]).
+   */
+  def isAutoFilledGeneratedColumn(attr: Attribute): Boolean = {
+    attr.metadata.contains(AUTO_FILLED_GENERATED_COLUMN_METADATA_KEY)
+  }
+
+  private def withMetadataEntry(
+      attr: AttributeReference,
+      key: String,
+      value: String): AttributeReference = {
+    val metadata = new MetadataBuilder()
+      .withMetadata(attr.metadata)
+      .putString(key, value)
+      .build()
+    attr.withMetadata(metadata).asInstanceOf[AttributeReference]
+  }
 }
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
index 4fd85ef4923d..a79e53ab85ef 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
@@ -248,7 +248,9 @@ package object util extends Logging {
     
FileSourceGeneratedMetadataStructField.FILE_SOURCE_GENERATED_METADATA_COL_ATTR_KEY,
     MetadataColumn.PRESERVE_ON_DELETE,
     MetadataColumn.PRESERVE_ON_UPDATE,
-    MetadataColumn.PRESERVE_ON_REINSERT
+    MetadataColumn.PRESERVE_ON_REINSERT,
+    GeneratedColumn.GENERATION_EXPRESSION_METADATA_KEY,
+    GeneratedColumn.AUTO_FILLED_GENERATED_COLUMN_METADATA_KEY
   )
 
   def removeInternalMetadata(schema: StructType, keepFieldIds: Boolean = 
false): StructType = {
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala
index a5f1ca7f1d28..37734a460986 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.catalog.{BucketSpec, 
ClusterBySpec}
 import org.apache.spark.sql.catalyst.expressions.AttributeReference
 import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
-import org.apache.spark.sql.catalyst.util.{quoteIfNeeded, quoteNameParts, 
QuotingUtils}
+import org.apache.spark.sql.catalyst.util.{quoteIfNeeded, quoteNameParts, 
removeInternalMetadata, QuotingUtils}
 import org.apache.spark.sql.connector.expressions.{BucketTransform, 
ClusterByTransform, FieldReference, IdentityTransform, LogicalExpressions, 
Transform}
 import org.apache.spark.sql.errors.{QueryCompilationErrors, 
QueryExecutionErrors}
 import org.apache.spark.sql.types.StructType
@@ -252,6 +252,15 @@ private[sql] object CatalogV2Implicits {
   implicit class ColumnsHelper(columns: Array[Column]) {
     def asSchema: StructType = CatalogV2Util.v2ColumnsToStructType(columns)
     def toAttributes: Seq[AttributeReference] = 
DataTypeUtils.toAttributes(asSchema)
+
+    /**
+     * Same as [[toAttributes]], but strips the internal metadata that must 
not surface in a
+     * relation's output, such as generation expressions. Column IDs are the 
exception: although
+     * the key is listed in INTERNAL_METADATA_KEYS so that other paths drop 
it, the column-ID
+     * feature deliberately surfaces field IDs on a relation's output.
+     */
+    def toOutputAttributes: Seq[AttributeReference] =
+      DataTypeUtils.toAttributes(removeInternalMetadata(asSchema, keepFieldIds 
= true))
   }
 
   def parseColumnPath(name: String): Seq[String] = {
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala
index 2f139393ade3..8015d3d0f9cf 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala
@@ -25,7 +25,6 @@ import 
org.apache.spark.sql.catalyst.analysis.NamedStreamingRelation
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, 
UnresolvedDataSource}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
-import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes
 import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
 import org.apache.spark.sql.classic.SparkSession
 import org.apache.spark.sql.connector.catalog.{SupportsRead, TableProvider}
@@ -112,7 +111,7 @@ class ResolveDataSource(sparkSession: SparkSession) extends 
Rule[LogicalPlan] {
               import 
org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
               StreamingRelationV2(
                   Some(provider), source, table, dsOptions,
-                  toAttributes(table.columns.asSchema), None, None, v1Relation,
+                  table.columns.toOutputAttributes, None, None, v1Relation,
                   v1DataSource.streamingSourceIdentifyingName)
 
             // fallback to v1
@@ -173,7 +172,7 @@ class ResolveDataSource(sparkSession: SparkSession) extends 
Rule[LogicalPlan] {
               import 
org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
               StreamingRelationV2(
                   Some(provider), source, table, dsOptions,
-                  toAttributes(table.columns.asSchema), None, None, v1Relation,
+                  table.columns.toOutputAttributes, None, None, v1Relation,
                   v1DataSource.streamingSourceIdentifyingName)
 
             // fallback to v1
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 81b2ea59cad9..c963d91005f1 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -18,10 +18,12 @@
 package org.apache.spark.sql.connector
 
 import org.apache.spark.SparkRuntimeException
-import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.{AnalysisException, DataFrame, QueryTest, Row}
 import org.apache.spark.sql.catalyst.QueryPlanningTracker
 import org.apache.spark.sql.catalyst.expressions.CheckInvariant
-import 
org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog
+import org.apache.spark.sql.catalyst.util.GeneratedColumn
+import org.apache.spark.sql.connector.catalog.{Identifier, 
InMemoryRowLevelOperationTableCatalog}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
 import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
 import org.apache.spark.sql.internal.SQLConf
 
@@ -97,6 +99,17 @@ class GeneratedColumnWriteSuite extends QueryTest with 
DatasourceV2SQLBase {
     }
   }
 
+  private def generationExpressionOf(table: String, column: String): 
Option[String] = {
+    val loaded = 
catalog("testcat").asTableCatalog.loadTable(Identifier.of(Array(), table))
+    Option(loaded.columns().find(_.name == column).get.generationExpression())
+  }
+
+  private def assertNoGenerationMetadata(df: DataFrame, column: String): Unit 
= {
+    val metadata = df.schema(column).metadata
+    
assert(!metadata.contains(GeneratedColumn.GENERATION_EXPRESSION_METADATA_KEY),
+      s"generation expression leaked into the schema of $column: 
${metadata.json}")
+  }
+
   testGeneratedColumnWrite("append_data_v2") { table =>
     import testImplicits._
     Seq(Tuple5(1L, "foo", "2020-10-11 12:30:30", 100, "2020-11-12"))
@@ -979,6 +992,68 @@ class GeneratedColumnWriteSuite extends QueryTest with 
DatasourceV2SQLBase {
     }
   }
 
+  test("generation expression is not exposed in the read schema") {
+    val tblName = "my_tab"
+    withTable(s"testcat.$tblName") {
+      sql(s"""CREATE TABLE testcat.$tblName(
+             |  id INT,
+             |  doubled INT GENERATED ALWAYS AS (id * 2)
+             |) USING foo""".stripMargin)
+      // The table still declares the generated column; only the internal 
metadata is hidden.
+      assert(generationExpressionOf(tblName, "doubled").contains("id * 2"))
+      assertNoGenerationMetadata(spark.table(s"testcat.$tblName"), "doubled")
+      assertNoGenerationMetadata(spark.read.table(s"testcat.$tblName"), 
"doubled")
+      assertNoGenerationMetadata(sql(s"SELECT * FROM testcat.$tblName"), 
"doubled")
+      assertNoGenerationMetadata(spark.readStream.table(s"testcat.$tblName"), 
"doubled")
+    }
+  }
+
+  test("CTAS from a table with generated columns does not create generated 
columns") {
+    val srcName = "src_tab"
+    val dstName = "dst_tab"
+    withTable(s"testcat.$srcName", s"testcat.$dstName") {
+      sql(s"""CREATE TABLE testcat.$srcName(
+             |  id INT,
+             |  doubled INT GENERATED ALWAYS AS (id * 2)
+             |) USING foo""".stripMargin)
+      sql(s"INSERT INTO testcat.$srcName (id) VALUES (1)")
+      sql(s"CREATE TABLE testcat.$dstName USING foo AS SELECT * FROM 
testcat.$srcName")
+      // The query output is plain data, so the new table must not inherit the 
generated column.
+      assert(generationExpressionOf(dstName, "doubled").isEmpty)
+      // Confirms it behaves as an ordinary column: a value the expression 
would not produce is
+      // written through instead of failing a generated column constraint.
+      sql(s"INSERT INTO testcat.$dstName VALUES (5, 999)")
+      checkAnswer(spark.table(s"testcat.$dstName"), Row(1, 2) :: Row(5, 999) 
:: Nil)
+    }
+  }
+
+  test("streaming write to a new table does not create generated columns") {
+    val srcName = "src_tab"
+    val dstName = "dst_tab"
+    withTable(s"testcat.$srcName", s"testcat.$dstName") {
+      withTempDir { checkpointDir =>
+        sql(s"""CREATE TABLE testcat.$srcName(
+               |  id INT,
+               |  doubled INT GENERATED ALWAYS AS (id * 2)
+               |) USING foo""".stripMargin)
+        sql(s"INSERT INTO testcat.$srcName (id) VALUES (1)")
+        // toTable creates the target from the streaming DataFrame's schema, 
which must not carry
+        // the source's generation expression over: a new table with a 
generated column would make
+        // this very write unsupported.
+        val query = spark.readStream.table(s"testcat.$srcName").writeStream
+          .option("checkpointLocation", checkpointDir.getAbsolutePath)
+          .toTable(s"testcat.$dstName")
+        try {
+          query.processAllAvailable()
+        } finally {
+          query.stop()
+        }
+        assert(generationExpressionOf(dstName, "doubled").isEmpty)
+        checkAnswer(spark.table(s"testcat.$dstName"), Row(1, 2))
+      }
+    }
+  }
+
   test("streaming write with generated columns is blocked") {
     import testImplicits._
     val tblName = "my_tab"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to