Aayush7352 opened a new pull request, #57030:
URL: https://github.com/apache/spark/pull/57030
### What changes were proposed in this pull request?
This PR fixes Parquet MIN/MAX aggregate push-down so that an all-null row
group contributes `NULL` (i.e. nothing) to the aggregate instead of failing the
query, and an entirely-null column yields `NULL` as MIN/MAX results:
- `ParquetUtils.getCurrentBlockMaxOrMin` now distinguishes two cases when
column statistics have no min/max (`!statistics.hasNonNullValue`):
- **All-null block** (`statistics.isNumNullsSet && statistics.getNumNulls
== valueCount`): returns `null`, meaning the block contributes nothing to
MIN/MAX - mirroring `OrcUtils.getMinMaxFromColumnStatistics`, which returns
`null` when `getNumberOfValues == 0`.
- **Genuinely missing statistics**: still throws
`PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX` (e.g. files written with
`parquet.column.statistics.enabled=false`), preserving the behavior guarded by
the SPARK-57746 test.
- The MIN/MAX accumulation loop in `getPushedDownAggResult` skips `null`
block contributions, and appends `null` (instead of the `None` sentinel) when
no block contributed a value.
- `createAggInternalRowFromFooter` leaves the corresponding result field
`NULL` when the pushed-down value is `null` (fields of `SpecificInternalRow`
start as `NULL`).
### Why are the changes needed?
`MIN`/`MAX` over a column whose values are all `NULL` must return `NULL` per
SQL semantics. Before this change, Parquet aggregate push-down threw an error
as soon as **any** row group of the column was all-null:
```scala
spark.range(10).selectExpr("CAST(NULL AS LONG) AS v").write.parquet("/tmp/t")
spark.conf.set("spark.sql.parquet.aggregatePushdown", true)
spark.read.parquet("/tmp/t").selectExpr("MIN(v)").show()
```
fails with:
```
[PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX] ... No min/max value
found in the column statistics. SQLSTATE: 0A000
```
while the same query succeeds and returns `NULL` with push-down disabled,
and with the ORC data source (which already treats an all-null stripe as "no
contribution"). Fixes SPARK-57739.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.parquet.aggregatePushdown=true`, MIN/MAX aggregation
over a Parquet column containing an all-null row group previously failed with
`PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX`; it now returns the
correct result (`NULL` when the whole column is null), consistent with
push-down disabled and with ORC. Queries on files with genuinely missing
statistics still fail with the same error as before.
### How was this patch tested?
Two new tests in `FileSourceAggregatePushDownSuite`, which run across
Parquet V1/V2 and ORC V1/V2:
- MIN/MAX of an all-null column return `NULL` (alongside a non-null column,
COUNT unchanged).
- An all-null file and a non-null file in the same table combine correctly
(all-null block contributes nothing).
Both tests fail on master with
`PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX` in the Parquet suites and
pass with this change:
```
build/sbt 'sql/testOnly *AggregatePushDownSuite'
...
Tests: succeeded 105, failed 0, canceled 0, ignored 0, pending 0
```
This includes the existing SPARK-57746 test verifying that files written
with statistics disabled still raise `NO_MIN_MAX`/`NO_NUM_NULLS`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
