szehon-ho opened a new pull request, #56002: URL: https://github.com/apache/spark/pull/56002
## Summary Backport of #55427 to `branch-4.2`. Adds support for `INSERT INTO ... WITH SCHEMA EVOLUTION` to fill missing nested struct fields with null (or column defaults) when the source has fewer fields than the target table, mirroring existing `MERGE INTO` behavior gated by `spark.sql.mergeNestedTypeCoercion.enabled`. Key changes: - New config: `spark.sql.insertNestedTypeCoercion.enabled` (internal, default `false`) - Refactor `TableOutputResolver.resolveOutputColumns` to use `DefaultValueFillMode` (`NONE`, `FILL`, `RECURSE`) - Enable `RECURSE` mode for V2 inserts when schema evolution and the coercion flag are both enabled - 17 new tests in `InsertIntoSchemaEvolutionTests` (via `InsertIntoTests.scala`) ## Why are the changes needed? `MERGE INTO` already supports nested type coercion when the source has fewer struct fields than the target. `INSERT INTO WITH SCHEMA EVOLUTION` lacked this capability, causing errors for legitimate schema-evolution workflows where older sources omit newer nested fields. ## Does this PR introduce _any_ user-facing change? Yes. When `spark.sql.insertNestedTypeCoercion.enabled` is set to `true` (default `false`), `INSERT INTO ... WITH SCHEMA EVOLUTION` fills missing nested struct fields with null instead of failing. ## How was this patch tested? Cherry-picked from #55427 onto current `branch-4.2` (`bd8872a0cc7`) with clean cherry-picks (no conflicts). Original PR test plan: - Added comprehensive positive/negative tests in `InsertIntoTests.scala` - All matched tests passed on master ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (Claude Opus 4) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
