[
https://issues.apache.org/jira/browse/GOBBLIN-1505?focusedWorklogId=641359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641359
]
ASF GitHub Bot logged work on GOBBLIN-1505:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Aug/21 22:27
Start Date: 24/Aug/21 22:27
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r695254629
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent
deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME,
properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits =
Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName =
MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+
properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+
properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ SortedMap<String, Gauge> existingGauges =
this.getMetricContext().getGauges(
+ (name, metric) -> name.equals(workUnitCountName));
Review comment:
We are doing a full scan of metrics list here, so if there are 100k of
them, it can have a performance impact. You can run a profiler on a large
deployment to see if a lot of time is spent here. If it is, then we can add a
method to check metric existence by name without scanning. Don't see such
method in the current library.
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent
deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME,
properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits =
Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
There is a quite sizable chunk of logic in this "case" branch, you can
move it to a separate method to make the current one smaller and easier to
reason about.
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -64,6 +72,8 @@
@Getter
private Meter messageParseFailures;
+ private final HashMap<String, Long> flowNameGroupToWorkUnitCount;
Review comment:
I see that other fields in this class are ThreadLocal, which may
indicate that this class can be called from multiple threads. Also, I'm pretty
sure that the lambda " () ->
this.flowNameGroupToWorkUnitCount.get(workUnitCountName)" below will be called
by metrics library from a separate thread.
If a collection is accessed by multiple threads at the same time, you'll get
occasional data corruption and strange exceptions.
You can use a thread-safe collection here instead from Apache commons
library or Guava.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 641359)
Time Spent: 7.5h (was: 7h 20m)
> Add Metric for JobSize (in bytes/records) per Flow
> --------------------------------------------------
>
> Key: GOBBLIN-1505
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1505
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> From a user/platform perspective, we want to see how many workunits are being
> created for each flow. The metric of workunit count will be emitted in the
> form <flowgroup>.<flowname>.jobSize
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
