fuwhu opened a new pull request #27213: [SPARK-30516][SQL] statistic estimation of FileScan should take partitionFilters and partition number into account URL: https://github.com/apache/spark/pull/27213 ### What changes were proposed in this pull request? Refine FileScan.estimateStatistics to take partitionFilters and MAX_PARTITION_NUMBER_FOR_STATS_CALCULATION_VIA_FS into account. ### Why are the changes needed? Currently, FileScan.estimateStatistics does not take partitionFilters and partition number into account, which may lead to bigger sizeInBytes. It should be reasonable to change it to involve partitionFilters and partition number when estimating the statistics. ### Does this PR introduce any user-facing change? no ### How was this patch tested? existing unit tests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
