[
https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021611#comment-15021611
]
Chengxiang Li commented on HIVE-12466:
--------------------------------------
[~xuefuz], due to the limitation of Spark accumulator, the `SparkCounter` has
to register the counter name before the job execution. The error message shows
that specified counter name is not registered before. In default, all the
default spark counters are collected with `SparkTask::getCounterPrefixes()`,
`RECORDS_OUT_0`, `RECORDS_OUT_1_default.tmp_tmp` and
`RECORDS_OUT_1_default.test_table` are not included, seems the counter logic
changes in `ReduceSinkOperator` and 'FileSinkOperator', we need to update the
logic of `SparkTask::getOperatorCounters`.
> SparkCounter not initialized error
> ----------------------------------
>
> Key: HIVE-12466
> URL: https://issues.apache.org/jira/browse/HIVE-12466
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Rui Li
> Assignee: Xuefu Zhang
>
> During a query, lots of the following error found in executor's log:
> {noformat}
> 03:47:28.759 [Executor task launch worker-0] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE, RECORDS_OUT_0]
> has not initialized before.
> 03:47:28.762 [Executor task launch worker-1] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE, RECORDS_OUT_0]
> has not initialized before.
> 03:47:30.707 [Executor task launch worker-1] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE,
> RECORDS_OUT_1_default.tmp_tmp] has not initialized before.
> 03:47:33.385 [Executor task launch worker-1] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE,
> RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:33.388 [Executor task launch worker-0] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE,
> RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:33.495 [Executor task launch worker-0] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE,
> RECORDS_OUT_1_default.test_table] has not initialized before.
> 03:47:35.141 [Executor task launch worker-1] ERROR
> org.apache.hive.spark.counter.SparkCounters - counter[HIVE,
> RECORDS_OUT_1_default.test_table] has not initialized before.
> ...........
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
