[ 
https://issues.apache.org/jira/browse/GOBBLIN-2029?focusedWorklogId=912103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-912103
 ]

ASF GitHub Bot logged work on GOBBLIN-2029:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 28/Mar/24 19:46
            Start Date: 28/Mar/24 19:46
    Worklog Time Spent: 10m 
      Work Description: phet commented on code in PR #3906:
URL: https://github.com/apache/gobblin/pull/3906#discussion_r1543543370


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProc.java:
##########
@@ -70,41 +51,33 @@ public class LaunchDagProc extends 
DagProc<Optional<Dag<JobExecutionPlan>>, Opti
         
metricContext.newContextAwareGauge(ServiceMetricNames.FLOW_ORCHESTRATION_DELAY, 
orchestrationDelayCounter::get));
   }
 
-  @Override
-  protected DagManager.DagId getDagId() {
-    return this.launchDagTask.getDagId();
+  public LaunchDagProc(LaunchDagTask dagTask, FlowCompilationValidationHelper 
flowCompilationValidationHelper) {
+    super(dagTask);
+    this.flowCompilationValidationHelper = flowCompilationValidationHelper;
   }
 
   @Override
   protected Optional<Dag<JobExecutionPlan>> initialize(DagManagementStateStore 
dagManagementStateStore)
       throws IOException {
     try {
-      DagActionStore.DagAction dagAction = this.launchDagTask.getDagAction();
-      FlowSpec flowSpec = loadFlowSpec(dagManagementStateStore, dagAction);
-      flowSpec.addProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY, 
dagAction.getFlowExecutionId());
+      FlowSpec flowSpec = 
dagManagementStateStore.getFlowSpec(FlowSpec.Utils.createFlowSpecUri(getDagId().getFlowId()));
+      flowSpec.addProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY, 
getDagId().getFlowExecutionId());
       return 
this.flowCompilationValidationHelper.createExecutionPlanIfValid(flowSpec).toJavaUtil();
     } catch (URISyntaxException | SpecNotFoundException | InterruptedException 
e) {
       throw new RuntimeException(e);
     }
   }
 
-  private FlowSpec loadFlowSpec(DagManagementStateStore 
dagManagementStateStore, DagActionStore.DagAction dagAction)
-      throws URISyntaxException, SpecNotFoundException {
-    URI flowUri = FlowSpec.Utils.createFlowSpecUri(dagAction.getFlowId());
-    return dagManagementStateStore.getFlowSpec(flowUri);
-  }
-
   @Override
-  protected Optional<Dag<JobExecutionPlan>> act(DagManagementStateStore 
dagManagementStateStore, Optional<Dag<JobExecutionPlan>> dag)
+  protected void act(DagManagementStateStore dagManagementStateStore, 
Optional<Dag<JobExecutionPlan>> dag)
       throws IOException {
     if (!dag.isPresent()) {
       log.warn("Dag with id " + getDagId() + " could not be compiled.");
       // todo - add metrics
-      return Optional.empty();
+    } else {
+      submitNextNodes(dagManagementStateStore, dag.get());
+      orchestrationDelayCounter.set(System.currentTimeMillis() - 
DagManagerUtils.getFlowExecId(dag.get()));

Review Comment:
   I realize this is purely for a metric, so not as relevant as [this 
comment](https://github.com/apache/gobblin/pull/3899#discussion_r1540638957) 
about hard-coded "clocks".
   
   if you do wish to address this case, however, I'd advise to define `long 
DMSS::getCurrentTimeMillis()`.
   
   any test wanting to insist on a known clock could then override DMSS and 
override that method



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceGuiceModule.java:
##########
@@ -38,11 +38,7 @@
 import javax.inject.Singleton;
 
 import org.apache.gobblin.restli.EmbeddedRestliServer;
-import org.apache.gobblin.service.modules.orchestration.DagActionStore;
 import org.apache.gobblin.runtime.api.GobblinInstanceEnvironment;
-import 
org.apache.gobblin.service.modules.orchestration.MultiActiveLeaseArbiter;
-import 
org.apache.gobblin.service.modules.orchestration.MysqlMultiActiveLeaseArbiter;
-import org.apache.gobblin.service.modules.orchestration.MysqlDagActionStore;
 import org.apache.gobblin.runtime.instance.StandardGobblinInstanceLauncher;
 import org.apache.gobblin.runtime.spec_catalog.FlowCatalog;
 import org.apache.gobblin.runtime.spec_catalog.TopologyCatalog;

Review Comment:
   let's please better characterize import order for our code style.  the 
current two bullets are imprecise - 
https://gobblin.apache.org/docs/developer-guide/CodingStyle/
   
   we ought to specify:
   * where `javax.` goes
   * where `lombok.` goes
   * that `gobblin.` is pretty much in the past, and all replaced by 
`org.apache.gobblin.`
   * where unit-test-specific imports belong - personally I find it easiest to 
read `org.junit.`, `org.testng.`, `org.mockito.` etc. *following* all 
"non-test" imports (of what's being tested); I've observed others following 
this ordering as well
   * that `import static` belongs after `import` (seems common practice at 
least...)
   * putting `java.` and `javax.` first is typical and customary, but let's 
either justify or else drop the seemingly arbitrary guidance to deviate from 
alphabetical on:
       - `java.`, `org.` (non-gobblin), `com.`, `org.apache.gobblin.`
     rather than what would seem much easier for humans to scan:
       - `java.`, `com.`, `org.` (non-gobblin), `org.apache.gobblin.`



##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java:
##########
@@ -64,7 +64,10 @@
 import org.apache.gobblin.util.ConfigUtils;
 import org.apache.gobblin.util.retry.RetryerFactory;
 
-import static org.apache.gobblin.util.retry.RetryerFactory.*;
+import static org.apache.gobblin.util.retry.RetryerFactory.RETRY_INTERVAL_MS;
+import static org.apache.gobblin.util.retry.RetryerFactory.RETRY_TIME_OUT_MS;
+import static org.apache.gobblin.util.retry.RetryerFactory.RETRY_TYPE;
+import static org.apache.gobblin.util.retry.RetryerFactory.RetryType;

Review Comment:
   I don't see these newly used within (at least not showing in the diff)



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration.proc;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+
+import com.google.common.collect.Maps;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.metrics.event.TimingEvent;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecExecutor;
+import org.apache.gobblin.runtime.api.SpecProducer;
+import org.apache.gobblin.service.ExecutionStatus;
+import org.apache.gobblin.service.modules.flowgraph.Dag;
+import 
org.apache.gobblin.service.modules.orchestration.DagManagementStateStore;
+import org.apache.gobblin.service.modules.orchestration.DagManager;
+import org.apache.gobblin.service.modules.orchestration.DagManagerUtils;
+import org.apache.gobblin.service.modules.orchestration.TimingEventUtils;
+import org.apache.gobblin.service.modules.spec.JobExecutionPlan;
+
+
+/**
+ * A class to group together all the utility methods used by various dag procs.
+ */
+@Slf4j
+public class DagProcUtils {
+
+  /**
+   * - submits a {@link JobSpec} to a {@link SpecExecutor}
+   * - updates metrics
+   * - add updated dag node state to dagManagementStateStore
+   */
+  static void submitJobToExecutor(DagManagementStateStore 
dagManagementStateStore, Dag.DagNode<JobExecutionPlan> dagNode,

Review Comment:
   why package-protected rather than `public`?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration.proc;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+
+import com.google.common.collect.Maps;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.metrics.event.TimingEvent;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecExecutor;
+import org.apache.gobblin.runtime.api.SpecProducer;
+import org.apache.gobblin.service.ExecutionStatus;
+import org.apache.gobblin.service.modules.flowgraph.Dag;
+import 
org.apache.gobblin.service.modules.orchestration.DagManagementStateStore;
+import org.apache.gobblin.service.modules.orchestration.DagManager;
+import org.apache.gobblin.service.modules.orchestration.DagManagerUtils;
+import org.apache.gobblin.service.modules.orchestration.TimingEventUtils;
+import org.apache.gobblin.service.modules.spec.JobExecutionPlan;
+
+
+/**
+ * A class to group together all the utility methods used by various dag procs.

Review Comment:
   nit: "Utility methods for {@link DagProc} derived class impls"



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration.proc;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+
+import com.google.common.collect.Maps;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.metrics.event.TimingEvent;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecExecutor;
+import org.apache.gobblin.runtime.api.SpecProducer;
+import org.apache.gobblin.service.ExecutionStatus;
+import org.apache.gobblin.service.modules.flowgraph.Dag;
+import 
org.apache.gobblin.service.modules.orchestration.DagManagementStateStore;
+import org.apache.gobblin.service.modules.orchestration.DagManager;
+import org.apache.gobblin.service.modules.orchestration.DagManagerUtils;
+import org.apache.gobblin.service.modules.orchestration.TimingEventUtils;
+import org.apache.gobblin.service.modules.spec.JobExecutionPlan;
+
+
+/**
+ * A class to group together all the utility methods used by various dag procs.
+ */
+@Slf4j
+public class DagProcUtils {
+
+  /**
+   * - submits a {@link JobSpec} to a {@link SpecExecutor}
+   * - updates metrics
+   * - add updated dag node state to dagManagementStateStore
+   */
+  static void submitJobToExecutor(DagManagementStateStore 
dagManagementStateStore, Dag.DagNode<JobExecutionPlan> dagNode,
+      DagManager.DagId dagId) {
+    DagManagerUtils.incrementJobAttempt(dagNode);
+    JobExecutionPlan jobExecutionPlan = 
DagManagerUtils.getJobExecutionPlan(dagNode);
+    JobSpec jobSpec = DagManagerUtils.getJobSpec(dagNode);
+    Map<String, String> jobMetadata = 
TimingEventUtils.getJobMetadata(Maps.newHashMap(), jobExecutionPlan);
+
+    String specExecutorUri = DagManagerUtils.getSpecExecutorUri(dagNode);
+
+    // Run this spec on selected executor
+    SpecProducer<Spec> producer;
+    try {
+      producer = DagManagerUtils.getSpecProducer(dagNode);
+      TimingEvent jobOrchestrationTimer = 
DagProc.eventSubmitter.getTimingEvent(TimingEvent.LauncherTimings.JOB_ORCHESTRATED);
+
+      // Increment job count before submitting the job onto the spec producer, 
in case that throws an exception.
+      // By this point the quota is allocated, so it's imperative to increment 
as missing would introduce the potential to decrement below zero upon quota 
release.
+      // Quota release is guaranteed, despite failure, because exception 
handling within would mark the job FAILED.
+      // When the ensuing kafka message spurs DagManager processing, the quota 
is released and the counts decremented
+      // Ensure that we do not double increment for flows that are retried
+      if (DagManagerUtils.getJobExecutionPlan(dagNode).getCurrentAttempts() == 
1) {
+        
dagManagementStateStore.getDagManagerMetrics().incrementRunningJobMetrics(dagNode);
+      }
+      // Submit the job to the SpecProducer, which in turn performs the actual 
job submission to the SpecExecutor instance.
+      // The SpecProducer implementations submit the job to the underlying 
executor and return when the submission is complete,
+      // either successfully or unsuccessfully. To catch any exceptions in the 
job submission, the DagManagerThread
+      // blocks (by calling Future#get()) until the submission is completed.
+      dagManagementStateStore.tryAcquireQuota(Collections.singleton(dagNode));
+      Future<?> addSpecFuture = producer.addSpec(jobSpec);
+      // todo - we should add future.get() instead of the complete future into 
the JobExecutionPlan
+      
dagNode.getValue().setJobFuture(com.google.common.base.Optional.of(addSpecFuture));
+      addSpecFuture.get();
+      jobExecutionPlan.setExecutionStatus(ExecutionStatus.ORCHESTRATED);
+      jobMetadata.put(TimingEvent.METADATA_MESSAGE, 
producer.getExecutionLink(addSpecFuture, specExecutorUri));
+      // Add serialized job properties as part of the orchestrated job event 
metadata
+      jobMetadata.put(JobExecutionPlan.JOB_PROPS_KEY, 
dagNode.getValue().toString());
+      jobOrchestrationTimer.stop(jobMetadata);
+      log.info("Orchestrated job: {} on Executor: {}", 
DagManagerUtils.getFullyQualifiedJobName(dagNode), specExecutorUri);
+      
dagManagementStateStore.getDagManagerMetrics().incrementJobsSentToExecutor(dagNode);
+      dagManagementStateStore.addDagNodeState(dagNode, dagId);
+    } catch (Exception e) {
+      TimingEvent jobFailedTimer = 
DagProc.eventSubmitter.getTimingEvent(TimingEvent.LauncherTimings.JOB_FAILED);
+      String message = "Cannot submit job " + 
DagManagerUtils.getFullyQualifiedJobName(dagNode) + " on executor " + 
specExecutorUri;
+      log.error(message, e);
+      jobMetadata.put(TimingEvent.METADATA_MESSAGE, message + " due to " + 
e.getMessage());
+      if (jobFailedTimer != null) {
+        jobFailedTimer.stop(jobMetadata);
+      }
+      try {
+        // when there is no exception, quota will be released in job status 
monitor or re-evaluate dag proc
+        dagManagementStateStore.releaseQuota(dagNode);
+      } catch (IOException ex) {
+        log.error("Could not release quota while handling e", ex);
+      }
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static boolean hasRunningJobs(DagManager.DagId dagId, 
DagManagementStateStore dagManagementStateStore)
+      throws IOException {
+    List<Dag.DagNode<JobExecutionPlan>> dagNodes = 
dagManagementStateStore.getDagNodes(dagId);
+    return dagNodes != null && !dagNodes.isEmpty();
+  }

Review Comment:
   since the impl merely invokes a DMSS method and then analyzes the result, I 
wonder: would this better fit as a method on DMSS?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProc.java:
##########
@@ -19,47 +19,53 @@
 
 import java.io.IOException;
 
+import lombok.Getter;
+import lombok.RequiredArgsConstructor;
 import lombok.extern.slf4j.Slf4j;
 
 import org.apache.gobblin.annotation.Alpha;
 import org.apache.gobblin.configuration.State;
 import org.apache.gobblin.instrumented.Instrumented;
 import org.apache.gobblin.metrics.MetricContext;
 import org.apache.gobblin.metrics.event.EventSubmitter;
+import org.apache.gobblin.service.modules.flowgraph.DagNodeId;
 import 
org.apache.gobblin.service.modules.orchestration.DagManagementStateStore;
 import org.apache.gobblin.service.modules.orchestration.DagManager;
+import org.apache.gobblin.service.modules.orchestration.DagManagerUtils;
 import org.apache.gobblin.service.modules.orchestration.task.DagTask;
 
 
 /**
  * Responsible for performing the actual work for a given {@link DagTask} by 
first initializing its state, performing
- * actions based on the type of {@link DagTask} and finally submitting an 
event to the executor.
+ * actions based on the type of {@link DagTask}. Submitting events in time is 
found to be important in
+ * <a href="https://github.com/apache/gobblin/pull/3641";>PR#3641</a>, hence 
initialize and act methods submit events as
+ * they happen.
  */
 @Alpha
 @Slf4j
-public abstract class DagProc<S, T> {
+@RequiredArgsConstructor

Review Comment:
   we don't really want this ctor do we?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration.proc;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+
+import com.google.common.collect.Maps;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.metrics.event.TimingEvent;
+import org.apache.gobblin.runtime.api.JobSpec;
+import org.apache.gobblin.runtime.api.Spec;
+import org.apache.gobblin.runtime.api.SpecExecutor;
+import org.apache.gobblin.runtime.api.SpecProducer;
+import org.apache.gobblin.service.ExecutionStatus;
+import org.apache.gobblin.service.modules.flowgraph.Dag;
+import 
org.apache.gobblin.service.modules.orchestration.DagManagementStateStore;
+import org.apache.gobblin.service.modules.orchestration.DagManager;
+import org.apache.gobblin.service.modules.orchestration.DagManagerUtils;
+import org.apache.gobblin.service.modules.orchestration.TimingEventUtils;
+import org.apache.gobblin.service.modules.spec.JobExecutionPlan;
+
+
+/**
+ * A class to group together all the utility methods used by various dag procs.
+ */
+@Slf4j
+public class DagProcUtils {
+
+  /**
+   * - submits a {@link JobSpec} to a {@link SpecExecutor}
+   * - updates metrics

Review Comment:
   I'm somewhat weary of hard-coding metric activity within a static method... 
yet do wish to be pragmatic.
   
   in this case, why do you feel it works?  enumerating (in this javadoc) which 
metrics are updated within and how would really help weigh whether or not this 
is appropriate





Issue Time Tracking
-------------------

            Worklog Id:     (was: 912103)
    Remaining Estimate: 0h
            Time Spent: 10m

> code style changes DagProc and related classes
> ----------------------------------------------
>
>                 Key: GOBBLIN-2029
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-2029
>             Project: Apache Gobblin
>          Issue Type: Task
>            Reporter: Arjun Singh Bora
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to