[
https://issues.apache.org/jira/browse/SPARK-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell resolved SPARK-4092.
------------------------------------
Resolution: Fixed
Fix Version/s: 1.3.0
> Input metrics don't work for coalesce()'d RDD's
> -----------------------------------------------
>
> Key: SPARK-4092
> URL: https://issues.apache.org/jira/browse/SPARK-4092
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Patrick Wendell
> Assignee: Kostas Sakellis
> Priority: Critical
> Fix For: 1.3.0
>
>
> In every case where we set input metrics (from both Hadoop and block storage)
> we currently assume that exactly one input partition is computed within the
> task. This is not a correct assumption in the general case. The main example
> in the current API is coalesce(), but user-defined RDD's could also be
> affected.
> To deal with the most general case, we would need to support the notion of a
> single task having multiple input sources. A more surgical and less general
> fix is to simply go to HadoopRDD and check if there are already inputMetrics
> defined for the task with the same "type". If there are, then merge in the
> new data rather than blowing away the old one.
> This wouldn't cover case where, e.g. a single task has input from both
> on-disk and in-memory blocks. It _would_ cover the case where someone calls
> coalesce on a HadoopRDD... which is more common.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
