huangpengtao00-dotcom opened a new pull request, #59293:
URL: https://github.com/apache/spark/pull/59293
### What changes were proposed in this pull request?
`ResolveReferences` now expands stars in the aggregate expressions of a
`Pivot`, the same way it already does for `Aggregate`, so `count(*)` becomes
`count(1)` before `ResolvePivot` runs.
### Why are the changes needed?
In Spark 3.5, `functions.count` turned a star argument into
`Count(Literal(1))` itself. Since 4.0 it builds an unresolved `count(*)` call
and leaves the rewrite to the analyzer, which does it in
`ResolveReferences.expandStarExpression` (added in SPARK-42108).
`ResolveReferences` expands stars only for `Project`, `Filter`, `Aggregate` and
a few other operators, not for `Pivot`, so the star reaches `CheckAnalysis` and
fails. This is a regression from 3.5:
```python
df = spark.createDataFrame([(1, 'a'), (2, None)], "x int, s string")
df.groupBy('s').pivot('x', [1, 2]).agg(F.count('*')).show()
# 3.5: [Row(s=None, 1=None, 2=1), Row(s='a', 1=1, 2=None)]
# 4.0+: AnalysisException: [INVALID_USAGE_OF_STAR_OR_REGEX] Invalid usage
of '*' in Pivot.
```
SQL `PIVOT (count(*) FOR ...)` is not affected, because
`AstBuilder.visitFunctionCall` already rewrites `COUNT(*)` to `COUNT(1)` while
parsing.
### Does this PR introduce _any_ user-facing change?
Yes. A DataFrame pivot whose aggregate is `count("*")` works again and
returns the same result as in Spark 3.5, instead of failing with
`INVALID_USAGE_OF_STAR_OR_REGEX`.
### How was this patch tested?
- `AnalysisSuite`: a `Pivot` with an unresolved `count(*)` analyzes, and
every `Count` in the analyzed plan counts `Literal(1)`.
- `DataFramePivotSuite`: the reproduction above through
`groupBy().pivot().agg(count("*"))`.
Both tests fail with `INVALID_USAGE_OF_STAR_OR_REGEX` without the change.
`AnalysisSuite`, `DataFramePivotSuite` and the `pivot.sql` golden file pass.
```
build/sbt 'catalyst/testOnly *AnalysisSuite'
build/sbt 'sql/testOnly *DataFramePivotSuite'
build/sbt 'sql/testOnly *SQLQueryTestSuite -- -z pivot.sql'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
