szehon-ho opened a new pull request, #57441: URL: https://github.com/apache/spark/pull/57441
### What changes were proposed in this pull request? This PR updates `TableOutputResolver` so `INSERT WITH SCHEMA EVOLUTION` by position does not auto-fill trailing generated columns. If a by-position insert omits a trailing generated column, Spark now throws the same arity mismatch error as a regular by-position insert. The PR also adds coverage that: - by-position `INSERT WITH SCHEMA EVOLUTION` rejects an omitted trailing generated column; - by-name `INSERT WITH SCHEMA EVOLUTION` with an explicit column list still auto-fills the generated column. ### Why are the changes needed? By-position writes require values for all target columns. Generated columns are special because users can omit them only through by-name writes with an explicit column list, where Spark can unambiguously identify the stored input columns used by the generation expression. Without this change, the by-position schema-evolution path could auto-fill a trailing generated column, making it inconsistent with regular by-position inserts and with Delta behavior. ### Does this PR introduce _any_ user-facing change? Yes. In the current unreleased behavior, a by-position `INSERT WITH SCHEMA EVOLUTION` could omit a trailing generated column and have Spark auto-fill it. After this change, that insert fails with `INSERT_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS`. Users can still omit generated columns with a by-name insert and an explicit column list. ### How was this patch tested? Added a regression test in `GeneratedColumnWriteSuite`. Ran: ``` MAVEN_MIRROR_URL=https://maven-proxy.cloud.databricks.com build/sbt 'sql/testOnly org.apache.spark.sql.connector.GeneratedColumnWriteSuite -- -z "INSERT WITH SCHEMA EVOLUTION by position does not auto-fill trailing generated column"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
