[
https://issues.apache.org/jira/browse/SPARK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin resolved SPARK-10135.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
This has been fixed in 1.5.
> Percent of pruned partitions is shown wrong
> -------------------------------------------
>
> Key: SPARK-10135
> URL: https://issues.apache.org/jira/browse/SPARK-10135
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Romi Kuntsman
> Priority: Trivial
> Labels: easyfix
> Fix For: 1.5.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When reading partitioned Parquet in SparkSQL, an info message about the
> number of pruned partitions is displayed.
> Actual:
> "Selected 15 partitions out of 181, pruned -1106.6666666666667% partitions."
> Expected:
> "Selected 15 partitions out of 181, pruned 91.71270718232044% partitions."
> Fix: (i'm newbie here so please help make patch, thanks!)
> in DataSourceStrategy.scala in method apply()
> insted of:
> val percentPruned = (1 - total.toDouble / selected.toDouble) * 100
> should be:
> val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
