[
https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646213#comment-15646213
]
Rui Li commented on HIVE-15139:
-------------------------------
Hi [~ctang.ma], thanks for the review. Yeah your solution also works, like you
said, the stageId is not used when aggregating all the metrics. But it's also
no harm to pass the actual stageId which we have already stored in the map.
Why I remove the attempt number is because what we expect is just the stageId
down the road, e.g. in {{TaskInfo}}, {{MetricsCollection}} etc. That means
currently we don't differentiate multiple attempts for a single stage, in terms
of metrics.
In remote mode, we also only collect the stageId. You can refer to the
{{ClientListener}} in RemoteDriver. So this will make local mode consistent
with remote mode.
> HoS local mode fails with NumberFormatException
> -----------------------------------------------
>
> Key: HIVE-15139
> URL: https://issues.apache.org/jira/browse/HIVE-15139
> Project: Hive
> Issue Type: Bug
> Reporter: Rui Li
> Assignee: Rui Li
> Attachments: HIVE-15139.1.patch
>
>
> It's because we store {{stageId_attemptNum}} in JobMetricsListener but expect
> only {{stageId}} in LocalSparkJobStatus.
> {noformat}
> java.lang.NumberFormatException: For input string: "0_0"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.LocalSparkJobStatus.getSparkStatistics(LocalSparkJobStatus.java:146)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:104)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
