szehon-ho opened a new pull request, #57662:
URL: https://github.com/apache/spark/pull/57662
### What changes were proposed in this pull request?
A generated column's generation expression was carried as
`GENERATION_EXPRESSION` metadata on a relation's output attributes. This PR
treats that key as internal metadata, so it never reaches a relation's output,
and reworks the write path to take the expression from the table's V2 columns
instead:
- `GENERATION_EXPRESSION` is added to `INTERNAL_METADATA_KEYS`, so
`removeInternalMetadata` strips it. `DataSourceV2Relation.create` already ran
the table schema through that, so batch relations are clean; the
`StreamingRelationV2` paths built attributes directly from
`table.columns.toAttributes` and now use a new `toOutputAttributes`, which
applies the same strip (keeping field IDs, which the column-ID feature
deliberately exposes).
- `ResolveOutputRelation` attaches the expressions to the expected output
from `table.columns()` (the persisted source of truth) via
`GeneratedColumn.attachGenerationExpressions`, only for as long as resolving
the write takes, so `TableOutputResolver` can still auto-fill missing generated
columns.
- Instead of removing metadata from auto-filled columns to suppress
validation, the columns Spark computed are now marked with a new internal key,
`__auto_filled_generated_column`, and `ResolveTableConstraints` skips those.
Validation is therefore the default: any value that did not come from the
generation expression gets a `CheckInvariant`, as before.
### Why are the changes needed?
Because the expression rode along on the relation's output, it leaked in two
user-visible ways.
It showed up in a DataFrame's schema:
```scala
sql("CREATE TABLE testcat.t (id INT, doubled INT GENERATED ALWAYS AS (id *
2)) USING foo")
spark.table("testcat.t").schema("doubled").metadata
// {"GENERATION_EXPRESSION":"id * 2"}
```
And, since a table created from a query derives its columns from that
schema, the generated column was promoted back into a real generated column on
the new table:
```scala
sql("CREATE TABLE testcat.dst USING foo AS SELECT * FROM testcat.t")
// dst.doubled is a generated column copied from the source
spark.readStream.table("testcat.t").writeStream.toTable("testcat.dst2")
// fails: the target is created with a generated column, and streaming
writes to
// tables with generated columns are unsupported
```
Generated column values on writes were added recently (SPARK-57644) and are
not in any release, so this corrects the behavior of an unreleased feature.
### Does this PR introduce _any_ user-facing change?
Yes, within unreleased branches only. `GENERATION_EXPRESSION` no longer
appears in the metadata of a DataFrame's schema (batch reads, `readStream`, and
`SELECT`), and `CREATE TABLE AS SELECT` / `writeStream.toTable` now create
ordinary columns rather than inheriting the source's generated columns.
Auto-filling and validation of generated columns on writes to an existing table
are unchanged.
### How was this patch tested?
`GeneratedColumnWriteSuite` (72 tests, all passing), with three new cases:
- generation expression is not exposed in the read schema, covering
`spark.table`, `spark.read.table`, `SELECT`, and `readStream.table`
- CTAS from a table with generated columns does not create generated
columns, and the resulting column behaves as an ordinary one
- streaming write to a new table does not create generated columns
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 2.5, Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
