xgbtxy opened a new pull request, #59763: URL: https://github.com/apache/spark/pull/59763
### What changes were proposed in this pull request? Return zero when a partially pushed COUNT has no partial results to combine. The final SUM of per-file counts is wrapped in `Coalesce(..., Literal(0L))`. The top-down rewrite matches only aggregates in the original ordinal map, so the newly introduced SUM is not looked up as an original aggregate. Add Parquet/ORC regressions with AQE and vectorized readers enabled/disabled, plus a Spark 4.4 migration note. The tests cover nonempty counts with NULL values, zero matching file partitions, MIN/MAX results, and grouped empty results. ### Why are the changes needed? [SPARK-60487](https://issues.apache.org/jira/browse/SPARK-60487) reports NULL instead of zero for COUNT when no file is read. A V2 file scan returns one partial count per file; if partition pruning selects no files, the final SUM has no input and yields NULL. That violates COUNT's non-null result and can change the result of SQL predicates using the count. The regression checks the InternalRow null bits before `checkAnswer`. A RowEncoder based on COUNT's non-nullable analyzed schema can read an unexpected NULL slot as a long zero, masking the defect in ordinary Scala `collect()`. On unchanged master, a diagnostic matrix produced raw NULL COUNT fields in all 32 combinations while ordinary Scala collect showed zero; this used the worktree's built SQL classes. The final tests directly assert raw nullability as well as row values. ### Does this PR introduce _any_ user-facing change? Yes. Global non-distinct COUNT queries with Parquet/ORC aggregate pushdown return a non-null zero when partition pruning selects no files. This concerns reads routed through Data Source V2 with the relevant aggregate-pushdown configuration enabled. The released-build reproduction used Spark 4.2.0; the native tests use master at `0aa4f9136bccc40856a5af688e3d16f8b38ebfc4`. ### How was this patch tested? - The final regression suite fails both Parquet/ORC tests on unchanged production (`Unexpected nullability at output column 0`) and passes with the patch. The tests are identical between the original and fixed runs. - Three related suites pass: **51/51**, with zero failures, errors, or skips: `ParquetV2AggregatePushDownSuite` (25), `OrcV2AggregatePushDownSuite` (24), and `FileSourceCountPushDownSuite` (2). - `JDBCV2Suite`'s `scan with aggregate push-down: partial push-down SUM, AVG, COUNT` passes (1/1) when selected directly with ScalaTest Runner `-s` / `-t`, using the Maven-produced classes and dependency classpath. This exercises embedded H2; no external database was used. - Official SBT `sql/scalastyle`: 846 main-source files checked, no findings. The added test also passes the same checker/configuration in a one-file scope. The full `sql/Test/scalastyle` task hits a parser error in unchanged `JDBCV2Suite.scala:160`; the identical error reproduces on a byte-exact copy from the base commit. That full test-source style task is not reported as passing. - `git diff --check` passes. ```sh ./build/mvn -pl sql/core -am -Dtest=none \ -DwildcardSuites=org.apache.spark.sql.execution.datasources.ParquetV2AggregatePushDownSuite,org.apache.spark.sql.execution.datasources.OrcV2AggregatePushDownSuite,org.apache.spark.sql.execution.datasources.FileSourceCountPushDownSuite \ test ``` Local native environment: Windows x64, Eclipse Temurin 17.0.20.1, Maven 3.9.16, Spark 5.0.0-SNAPSHOT. The released 4.2.0 probe and native master tests were run separately. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) This contribution is original work submitted under the project's Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
