[
https://issues.apache.org/jira/browse/GOBBLIN-1505?focusedWorklogId=637179&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-637179
]
ASF GitHub Bot logged work on GOBBLIN-1505:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Aug/21 23:04
Start Date: 11/Aug/21 23:04
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r687241795
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
+ String workUnitCountName =
MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY),
jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
Review comment:
FLOW_GROUP / FLOW_NAME can be null when the Gobblin job is executed
outside of Gobblin service (just as Azkaban flow). This also means that
"ServiceMetricNames.GOBBLIN_SERVICE_PREFIX" would not be applicable to such
jobs.
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
Review comment:
The base class has optional runtimeMetricContext. Can we use that? I
think it will add some extra tags to metrics to tell them apart across the job.
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
+ String workUnitCountName =
MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY),
jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge guage =
metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
Where do we emit other metrics for GaaS-triggered jobs - inside the job
itself or in the GaaS code?
I see that we are emitting CountEvent for work unit size earlier in the
code. I assume it will land in GaaS and be available for processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 637179)
Time Spent: 4h 20m (was: 4h 10m)
> Add Metric for WorkUnit count per Flow
> --------------------------------------
>
> Key: GOBBLIN-1505
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1505
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> From a user/platform perspective, we want to see how many workunits are being
> created for each flow. The metric of workunit count will be emitted in the
> form <flowgroup>.<flowname>.workunits
--
This message was sent by Atlassian Jira
(v8.3.4#803005)