[
https://issues.apache.org/jira/browse/GOBBLIN-1992?focusedWorklogId=902006&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-902006
]
ASF GitHub Bot logged work on GOBBLIN-1992:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jan/24 18:49
Start Date: 26/Jan/24 18:49
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#discussion_r1468017947
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
throw new RuntimeException(e);
}
}
+
+ private void setupMDCFromJobState(JobState jobState) {
Review Comment:
I expect this will be reused for other job launchers, even if we're merely
creating them during dev (like the GenWUs I'm about to release). to support
that, maybe situate in `Help` or `JobStateUtils`?
and since I believe this MDC stuff all happens transparently behind the
scenes, maybe less impl jargon in the name. e.g.
`propagateGaaSFlowExecutionContext` or `initFlowExecutionContext`?
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
throw new RuntimeException(e);
}
}
+
+ private void setupMDCFromJobState(JobState jobState) {
+ // TODO: log4j2 has better syntax around conditional logging such that the
key does not need to be included in the value
+ MDC.put(ConfigurationKeys.FLOW_NAME_KEY,
String.format("%s:%s",ConfigurationKeys.FLOW_NAME_KEY,
jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY, "NA")));
+ MDC.put(ConfigurationKeys.FLOW_GROUP_KEY,
String.format("%s:%s",ConfigurationKeys.FLOW_GROUP_KEY,
jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY, "NA")));
Review Comment:
nit: I don't like `NA`, since it's technically possible that could be a
valid gaas flow group. would this token work - `<<NOT SET>>`?
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share
context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs
+ */
+@Slf4j
+public class MDCContextPropagator implements ContextPropagator {
+
+ public String getName() {
+ return this.getClass().getName();
+ }
+
+ public Object getCurrentContext() {
Review Comment:
nit: I presume most, if not all of these methods implement declarations from
`ContextPropagator`. let's make clear which do by adding `@Override` to them
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
throw new RuntimeException(e);
}
}
+
+ private void setupMDCFromJobState(JobState jobState) {
+ // TODO: log4j2 has better syntax around conditional logging such that the
key does not need to be included in the value
Review Comment:
the key may actually assist machine processing. I'd suggest just changing
the key names, so they're actually ready to be fed directly back into the GaaS
REST endpoints, such as:
```
GET
/flowexecutions/(flowGroup:<groupName>,flowName:<flowName>,flowExecutionId:<flowExecutionId>)
```
hence `flowName`, rather than `flow.name`. bonus points if these can come
out comma-separated, and so ready to use w/ that REST API, rather than
space-separated as you showed in the example output.
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share
context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs
Review Comment:
not clear what "Temporal headers" are... is this about HTTP requests?
please explain
Issue Time Tracking
-------------------
Worklog Id: (was: 902006)
Time Spent: 0.5h (was: 20m)
> Allow per job level logging in Gobblin Temporal workflows
> ---------------------------------------------------------
>
> Key: GOBBLIN-1992
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1992
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: William Lo
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently the Temporal job context does not include the job metadata e.g.
> flow name, flow group, flow execution ID, which is critical for debugging
> workflows in the temporal cluster.
> We want to have per job logging so that it is easier to debug multiple
> Gobblin workflows on Temporal sent through GaaS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)