sarutak commented on code in PR #59125:
URL: https://github.com/apache/spark/pull/59125#discussion_r4142804678
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceAggregatePushDownSuite.scala:
##########
@@ -123,6 +123,35 @@ trait FileSourceAggregatePushDownSuite
}
}
+ test("SPARK-59607: COUNT on a null partition column is not overcounted with
push down") {
Review Comment:
The new test covers `SELECT COUNT(p)` for Parquet and ORC, which is good.
Since the same logic is duplicated on both engines, and the GROUP BY and
multi-partition-column paths reuse it, consider extending coverage to guard
against a regression in either copy:
- `SELECT p, COUNT(*), COUNT(p) FROM t GROUP BY p` with a null partition
present (verifies the null group key plus the `reOrderPartitionCol`/`JoinedRow`
path).
- multiple partition columns where only one is null (verifies the
`partitionSchema`-based `isNullAt(index)` indexing is not off by column).
(Note: the pre-existing partition tests, "Count(partition column)",
"partition group by", and "partition filter", all use non-null values like `id
% 3 as p`, so none of them currently exercise a null partition.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
