[
https://issues.apache.org/jira/browse/HIVE-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxiang Li updated HIVE-7552:
--------------------------------
Description:
MR/Tez use counters to collect job statistic information, while Spark does not
use accumulator to do the same thing. Instead, Spark store task metrics
information in TaskMetrics and send it back to scheduler. We could get spark
job statistic information through combine all TaskMetrics with SparkListener.
NO PRECOMMIT TESTS. This is for spark-branch only.
was:
MR/Tez use counters to collect job statistic information, while Spark has a
configurable metrics system based on the Coda Hale Metrics Library. We could
collect spark job statistic information through spark metrics system in hive
driver side.
NO PRECOMMIT TESTS. This is for spark-branch only.
> Collect spark job statistic through spark metrics[Spark Branch]
> ---------------------------------------------------------------
>
> Key: HIVE-7552
> URL: https://issues.apache.org/jira/browse/HIVE-7552
> Project: Hive
> Issue Type: New Feature
> Components: Spark
> Reporter: Chengxiang Li
>
> MR/Tez use counters to collect job statistic information, while Spark does
> not use accumulator to do the same thing. Instead, Spark store task metrics
> information in TaskMetrics and send it back to scheduler. We could get spark
> job statistic information through combine all TaskMetrics with SparkListener.
> NO PRECOMMIT TESTS. This is for spark-branch only.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
