n-young opened a new pull request, #60076: URL: https://github.com/apache/spark/pull/60076
### What changes were proposed in this pull request? Draft pending a public SPARK issue. Preserve grouping-column precedence when window extraction turns HAVING into a Filter. Keep complete grouping keys as helper aggregate outputs for late and correlated references, exclude those helpers from SELECT-alias lookup, and preserve the scope through analyzer rewrites. Reconcile subquery dependencies after correlated grouping expressions are mapped to aggregate outputs. `spark.sql.analyzer.havingWithWindowResolvesGroupingColumnFirst.enabled` is an internal configuration that defaults to `false`, including in tests. Regressions enable it explicitly and verify rollback with it disabled. The scope lookup recognizes grouping columns and matching resolved grouping expressions. Function and type normalization of unresolved computed grouping keys is unchanged. ### Why are the changes needed? Adding a window function can change which groups survive HAVING when an input grouping column shares a name with a SELECT alias. ### Does this PR introduce _any_ user-facing change? Yes, with the configuration enabled: ```sql SET spark.sql.analyzer.havingWithWindowResolvesGroupingColumnFirst.enabled=true; SELECT k + 10 AS k, avg(score) AS score, count(*) OVER () AS n FROM VALUES (1, 2), (1, 3), (2, 2), (2, 3) AS t(k, score) GROUP BY k HAVING k >= 2 AND score >= 2.5 ORDER BY 1; ``` The previous result is `(11, 2.5, 2), (12, 2.5, 2)`; the corrected result is `(12, 2.5, 1)`. HAVING filters the input grouping column before the window count. Aggregate-alias types and ambiguity fallback are preserved. ### How was this patch tested? All 149 tests passed (2 HAVING golden tests, 88 analyzer tests, 59 window tests), with catalyst scalastyle reporting zero errors and warnings: ```text build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -t having.sql -t having.sql_analyzer_test' build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.analysis.AnalysisSuite' build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameWindowFunctionsSuite' build/sbt catalyst/scalastyle ``` Both HAVING golden tests fail when the scope fix is temporarily removed, reproducing the incorrect rows and window count above. The fixed source was restored before the final passing run. The disabled-configuration case explicitly preserves legacy behavior. Fourteen small synthetic regression queries and generated execution and analyzed-plan goldens cover grouping precedence, computed and nested keys, alias fallback, SQL UDF expansion, lambda captures, correlated references, and rollback. Expected rows and window counts change because HAVING now filters the intended groups; helper columns do not appear in the final SELECT output. Standard dual-run coverage does not exercise the single-pass analyzer for these queries: its guard excludes window expressions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
