HyukjinKwon commented on PR #59125:
URL: https://github.com/apache/spark/pull/59125#issuecomment-5969764806
Agreed that this is a genuine correctness issue. The
__HIVE_DEFAULT_PARTITION__ directory represents a real SQL NULL, so before this
change the pushed COUNT(partitionColumn) counted those rows and diverged from
the non-pushed result. The fix contributes 0 for a null partition value in both
the Parquet and ORC footer paths, restoring parity with the non-pushed count,
while COUNT(*), data-column counts, and MIN/MAX stay unchanged. It is worth
fixing as a correctness bug even though aggregate push-down is off by default.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:5884906288","thread_id":"conversation:5884906288","verdict_sha256":"706e13364546ed6082b639cb4821e9c3ad298ec3bb7a0bce1e7b89af95540060"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
