[ 
https://issues.apache.org/jira/browse/SPARK-30516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hu Fuwang updated SPARK-30516:
------------------------------
    Description: Currently, FileScan.estimateStatistics will not take 
partitionFilters into account, which may lead to bigger sizeInBytes. It should 
be reasonable to change it to involve partitionFilters and partition numbers 
when estimating the statistics.  (was: Currently, FileScan.estimateStatistics 
will not take partitionFilters into account, which may lead to bigger 
sizeInBytes.

It should be reasonable to change it to involve partitionFilters and partition 
numbers when estimating the statistics.)

> FileScan.estimateStatistics does not take partitionFilters and partition 
> number into account
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-30516
>                 URL: https://issues.apache.org/jira/browse/SPARK-30516
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Hu Fuwang
>            Priority: Major
>
> Currently, FileScan.estimateStatistics will not take partitionFilters into 
> account, which may lead to bigger sizeInBytes. It should be reasonable to 
> change it to involve partitionFilters and partition numbers when estimating 
> the statistics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to