[
https://issues.apache.org/jira/browse/PIG-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625669#comment-14625669
]
Mohit Sabharwal commented on PIG-4633:
--------------------------------------
Thanks, [~kexianda]. Just curious - how did you discover this ? Was there an
exception in the log ... or was some unit test failing ?
> Update hadoop version to enable Spark output statistics
> -------------------------------------------------------
>
> Key: PIG-4633
> URL: https://issues.apache.org/jira/browse/PIG-4633
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: kexianda
> Assignee: kexianda
> Fix For: spark-branch
>
> Attachments: PIG-4633.patch
>
>
> Spark support output statistics from 1.3.0 ([SPARK-3179. Add task
> OutputMetrics|https://issues.apache.org/jira/browse/SPARK-3179])
> {code:title=SparkHadoopUtil.scala|borderStyle=solid}
> stats.map(Utils.invoke(classOf[Statistics], _, "getThreadStatistics"))
> {code}
> Spark invoke hadoop's function getThreadStatistics. But, this method was
> added into hadoop from version 2.5.0
> ([HADOOP-10688|https://issues.apache.org/jira/browse/HADOOP-10688])
> The version of hadoop in ivy/libraries.properties should be 2.5.0 +
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
