Hi Xiuzhu, If I understand correctly, you're trying to get counter info in the code block(`TaskState#toTaskExecutionInfo`) when the job is running in mapreduce mode.
The answer is: You can't get task metrics from the driver in mapreduce mode. Because, in mapreduce mode, `Task` runs on a mapper, which is usually a different machine or jvm from the driver. You can't read the task metrics set on a different jvm. In standalone mode, `Task` runs on the same machine or jvm as the driver, that's why you can read the value. However, if your goal is to fetch task or job metrics, the right way is: 1. configure the job to emit metrics to a metrics store, that's why I mentioned `reporter` in previous reply. For more information, you can read https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/ Quick Start - Gobblin Documentation<https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/> Gobblin Metrics is a metrics library for emitting metrics and events instrumenting java applications. Metrics and events are easy to use and enriched with tags. gobblin.readthedocs.io 2. find the metrics you want in the store. Thanks, Zhixiong ________________________________ From: Zhang, Xiuzhu(AWF) <[email protected]> Sent: Monday, June 11, 2018 6:58 PM To: [email protected]; [email protected] Subject: RE: Gobblin Metrics Hi Zhixiong, I haven’t use configure reporter and haven’t sought anything exceptions. Looks like the TaskState.java responsible to collect metrics of counter type as follows: // Add task metrics TaskMetrics taskMetrics = TaskMetrics.get(this); MetricArray metricArray = new MetricArray(); for (Map.Entry<String, ? extends com.codahale.metrics.Metric> entry : taskMetrics.getMetricContext().getCounters() .entrySet()) { Metric counter = new Metric(); counter.setGroup(MetricGroup.TASK.name()); counter.setName(entry.getKey()); counter.setType(MetricTypeEnum.valueOf(GobblinMetrics.MetricType.COUNTER.name())); counter.setValue(Long.toString(((Counter) entry.getValue()).getCount())); metricArray.add(counter); } Here the MetricContext object can be fetched but the counters size is 0, why the metrics added by Counter from MetricContext can’t be fetched on this place? Very strange the gobblin definite metrics also haven’t be fetched. But it is normal under standalone mode. If it is a bug or need to configure any other something for mapreduce mode? Thanks, Zhang Xiuzhu From: Zhixiong Chen [mailto:[email protected]] Sent: Tuesday, June 12, 2018 1:19 AM To: [email protected]; [email protected] Subject: Re: Gobblin Metrics Hi Xiuzhu, Which metric reporter are you using? Did you see exceptions about sending metrics in mapreduce mode? Zhixiong, ________________________________ From: Zhang, Xiuzhu(AWF) <[email protected]<mailto:[email protected]>> Sent: Monday, June 11, 2018 2:19 AM To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Gobblin Metrics Hi, Concerning gobblin metrics function, I am encounter a strange issue. I write some self-definite metrics using MetricContext object by InstrumentedExtractor and InstrumentedDataWriter. Running it at standalone and mapreduce mode, after job completed just only standalone have the metrics information in gobblin_task_metrics table I don’t understand why haven’t the metrics under mapreduce mode. Have anyone know where need to care about it? I use it as follows : xxxExtractor extends InstrumentedExtractor<xxx, xxx> { private MetricContext metricContext; public xxxExtractor(WorkUnitState state){ this.metricContext=this.getMetricContext(); } Private void xxx(){ Counter counter=metricContext.counter(“xx.xxx.xxx”); recordsCounter.inc(xxx); } } Thanks, Zhang Xiuzhu
