aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r687241795
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
+ String workUnitCountName =
MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY),
jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
Review comment:
FLOW_GROUP / FLOW_NAME can be null when the Gobblin job is executed
outside of Gobblin service (just as Azkaban flow). This also means that
"ServiceMetricNames.GOBBLIN_SERVICE_PREFIX" would not be applicable to such
jobs.
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
Review comment:
The base class has optional runtimeMetricContext. Can we use that? I
think it will add some extra tags to metrics to tell them apart across the job.
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits)
throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " +
countEventBuilder.getCount());
+ MetricContext
+ metricContext =
Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()),
getClass());
+ String workUnitCountName =
MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY),
jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge guage =
metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
Where do we emit other metrics for GaaS-triggered jobs - inside the job
itself or in the GaaS code?
I see that we are emitting CountEvent for work unit size earlier in the
code. I assume it will land in GaaS and be available for processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]