[
https://issues.apache.org/jira/browse/GOBBLIN-2014?focusedWorklogId=909293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-909293
]
ASF GitHub Bot logged work on GOBBLIN-2014:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Mar/24 22:17
Start Date: 11/Mar/24 22:17
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3894:
URL: https://github.com/apache/gobblin/pull/3894#discussion_r1520485487
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java:
##########
@@ -154,6 +154,7 @@ public org.apache.gobblin.configuration.State
parseJobStatus(GobblinTrackingEven
break;
case TimingEvent.FlowTimings.FLOW_SUCCEEDED:
case TimingEvent.LauncherTimings.JOB_SUCCEEDED:
+ case TimingEvent.LauncherTimings.JOB_COMPLETE:
Review Comment:
is the alt. to adding this in KafkaAvroJSM to instead have
`ExecuteGobblinWorkflowImpl` emit `JOB_SUCCEEDED`? if that's what
gobblin-on-MR emitted, I'd prefer to fix the new producer rather than change
the consumer side.
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ExecuteGobblinJobLauncher.java:
##########
@@ -30,6 +31,8 @@
import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.metrics.event.TimingEvent;
Review Comment:
aren't the hadoop imports to proceed gobblin ones?
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ExecuteGobblinJobLauncher.java:
##########
@@ -84,13 +87,46 @@ public void submitJob(List<WorkUnit> workunits) {
Config jobConfigWithOverrides =
applyJobLauncherOverrides(ConfigUtils.propertiesToConfig(this.jobProps));
Help.propagateGaaSFlowExecutionContext(this.jobProps);
-
- EventSubmitterContext eventSubmitterContext = new
EventSubmitterContext(this.eventSubmitter);
-
+ EventSubmitterContext eventSubmitterContext = new EventSubmitterContext(
+ addAdditionalMetadataTags(this.jobProps,
this.eventSubmitter.getTags()),
+ eventSubmitter.getNamespace()
+ );
int numWorkUnits =
workflow.execute(ConfigUtils.configToProperties(jobConfigWithOverrides),
eventSubmitterContext);
log.info("FINISHED - ExecuteGobblinWorkflow.execute = {}", numWorkUnits);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
+
+ private static List<Tag<?>> addAdditionalMetadataTags(Properties jobProps,
List<Tag<?>> additionalTags) {
Review Comment:
if this be core gobblin-on-temporal stuff, how about a method of
`EventSubmitterContext`?
e.g. call as:
```
EventSubmitterContext eventSubmitterContext = new
EventSubmitterContext(this.eventSubmitter).withAugmentedMetadata(this.jobProps);
```
(`withAugmentedMetadata` would return a new `EventSubmitterContext` that's
updated to have the new `Tag`s).
Issue Time Tracking
-------------------
Worklog Id: (was: 909293)
Time Spent: 0.5h (was: 20m)
> Add metadata tags for events sent from Temporal to be parseable in GaaS
> -----------------------------------------------------------------------
>
> Key: GOBBLIN-2014
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2014
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Gobblin's Temporal workflow launches through `ExecuteGobblinJobLauncher`, but
> the eventSubmitter created in that class does not have the necessary metadata
> needed in order to send events to GaaS.
> We want to populate the additional tags to read from the job props that
> contain the information needed for these events to be readable from GaaS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
