linhongliu-db opened a new pull request, #59884: URL: https://github.com/apache/spark/pull/59884
### What changes were proposed in this pull request? Derive VALUES output nullability from the coerced expressions. For eagerly materialized rows, restore non-nullability only for originally nonnullable columns whose already-evaluated values are all non-NULL. Collect original nullability in row order to avoid repeated indexing of List-backed rows. Tables deferred by CURRENT_LIKE expressions retain conservative post-coercion nullability. This change does not refine that deferred path. ### Why are the changes needed? With ANSI mode disabled and `spark.sql.legacy.decimal.retainFractionDigitsOnTruncate=true`, common-type coercion can overflow a non-null decimal into NULL. The old output metadata still marked the column nonnullable, allowing NULL-sensitive optimizations to produce incorrect results or generated projections to fail with a `NullPointerException`. For example, `SELECT count(c), count(*) FROM VALUES (99999999999999999999999999999999999999), (0.1) AS t(c)` should return `(1, 2)` under those settings. ### Does this PR introduce _any_ user-facing change? Yes. VALUES metadata accounts for NULLs introduced by coercion, correcting `IS NULL` filtering and `COUNT(column)`. Fitting eager values retain nonnullable metadata without evaluating expressions again. ### How was this patch tested? Catalyst regressions cover fitting values and overflow at three scales, originally nullable inputs, List-backed columns, single evaluation, deferred timestamps, ANSI errors, and runtime-evaluated VALUES projections. The SQL regression checks values, `IS NULL`, and counts with eager parsing and runtime expressions enabled and disabled. Full Catalyst suite passed: 26 tests, zero failures. The new regression cases failed against the unmodified production code. ```bash build/sbt 'catalyst/testOnly *ResolveInlineTablesSuite' build/sbt 'sql/testOnly *SQLQuerySuite -- -z "VALUES nullability accounts for decimal coercion"' ``` The focused SQL test passed and covers eager evaluation and runtime expressions each enabled and disabled. Before the fix, it failed with a NULL decimal projection error. Scala style checks passed for the Catalyst and SQL source and test code. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex CLI 0.164.0 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
