vboo123 opened a new pull request, #59125:
URL: https://github.com/apache/spark/pull/59125
### What changes were proposed in this pull request?
Fix aggregate push down so that `COUNT(partitionColumn)` does not count rows
in
a null partition (the `__HIVE_DEFAULT_PARTITION__` directory), matching the
non-pushed result.
The footer-based path computes a partition column's count from per-file row
counts. For a partition column it added the whole file row count and never
subtracted nulls, assuming a partition value is always non-null. A null
partition value breaks that assumption, so such a file must contribute 0 to
`COUNT(colName)`.
- `ParquetUtils.getPushedDownAggResult` now takes `partitionValues` and, for
a
partition column whose value is null (`partitionValues.isNullAt`),
contributes
0 instead of the block row count.
- `OrcUtils.createAggInternalRowFromFooter` (which already has
`partitionValues`)
returns `0L` for a null partition value instead of the file's total value
count.
Data columns, `COUNT(*)`, and `MIN`/`MAX` (never pushed on partition
columns) are
unchanged.
### Why are the changes needed?
Reported in SPARK-59607. On a table partitioned by `p` with partitions
`a` / `__HIVE_DEFAULT_PARTITION__` / `b`, one row each:
| Query | push down off | push down on (before) |
|--------------------|:-------------:|:---------------------:|
| `COUNT(*)` | 3 | 3 |
| `COUNT(v)` | 3 | 3 |
| `COUNT(p)` | 2 | **3** (wrong) |
| `COUNT(DISTINCT p)`| 2 | 2 |
`COUNT(p)` must skip rows where `p IS NULL`. The null partition
(`__HIVE_DEFAULT_PARTITION__`) is a genuine SQL NULL, so a pushed aggregate
must
return the same value as the non-pushed one.
### Does this PR introduce _any_ user-facing change?
Yes. With aggregate push down enabled (`spark.sql.parquet.aggregatePushdown`
/
`spark.sql.orc.aggregatePushdown`, both off by default) and a null partition
present, `COUNT(partitionColumn)` previously returned an inflated result; it
now
returns the correct value, equal to the non-pushed result.
### How was this patch tested?
New test in `FileSourceAggregatePushDownSuite` (shared, so it runs for
Parquet
and ORC across V1 and V2, both `vectorizedReader` settings). It builds
partitions
`a` / `__HIVE_DEFAULT_PARTITION__` / `b`, one row each, and asserts
`COUNT(*), COUNT(v), COUNT(p)` = `(3, 3, 2)` with push down both off and on.
### Was this patch authored or co-authored using generative AI tooling?
Generated using Kiro (Claude Opus 4.8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
