phet commented on code in PR #3890:
URL: https://github.com/apache/gobblin/pull/3890#discussion_r1522326805


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MultiActiveLeaseArbiter.java:
##########
@@ -82,15 +82,24 @@ abstract class LeaseAttemptStatus {}
 
   class NoLongerLeasingStatus extends LeaseAttemptStatus {}
 
+
+  /** `j.u.Function` variant for an operation that may @throw IOException */
+  @FunctionalInterface
+  interface FunctionMayThrowIO<T, R> {
+    R apply(T t) throws IOException;
+  }
+
   /*
   The participant calling this method acquired the lease for the event in 
question. `Flow action`'s flow execution id
   is the timestamp associated with the lease and the time the caller obtained 
the lease is stored within the
-  `leaseAcquisitionTimestamp` field.
+  `leaseAcquisitionTimestamp` field. `completeLeaseRunnable` is a reference to 
a method that will recordLeaseSuccess
+  for the current LeaseObtainedStatus from a caller without access to the 
{@link MultiActiveLeaseArbiter}
   */
   @Data
   class LeaseObtainedStatus extends LeaseAttemptStatus {
     private final DagActionStore.DagAction flowAction;
     private final long leaseAcquisitionTimestamp;
+    protected final FunctionMayThrowIO<LeaseObtainedStatus, Boolean> 
completeLeaseRunnable;

Review Comment:
   1. why `protected` (esp. when `@Data` already makes a `public 
getCompleteLeaseRunnable()`)?
   2. what's the advantage to this `Runnable` over just a field holding a MALA, 
so you could provide a method:
   ```
   public boolean completeLease() {
     return mala.recordLeaseSuccess(this);
   }
   ```
   
   (also tip: to keep that MALA reference inaccessible, even with `@Data`, put 
this at the field/member level: `@Getter(AccessLevel.NONE)`



##########
gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerService.java:
##########
@@ -60,20 +61,27 @@ public SchedulerService(Properties props) {
         Optional.of(PropertiesUtils.extractPropertiesWithPrefix(props, 
Optional.of("org.quartz."))));
   }
 
-  @Inject
   public SchedulerService(Config cfg) {
     this(cfg.hasPath(ConfigurationKeys.SCHEDULER_WAIT_FOR_JOB_COMPLETION_KEY) ?
          
cfg.getBoolean(ConfigurationKeys.SCHEDULER_WAIT_FOR_JOB_COMPLETION_KEY) :
          
Boolean.parseBoolean(ConfigurationKeys.DEFAULT_SCHEDULER_WAIT_FOR_JOB_COMPLETION),
          Optional.of(ConfigUtils.configToProperties(cfg, "org.quartz.")));
   }
 
+  @Inject
+  public SchedulerService(Config cfg, StdSchedulerFactory stdSchedulerFactory) 
{
+    this(cfg);
+    this.stdSchedulerFactory = stdSchedulerFactory;
+  }
+
   @Override protected void startUp() throws SchedulerException  {
-    StdSchedulerFactory schedulerFactory = new StdSchedulerFactory();
+    if (this.stdSchedulerFactory == null) {
+      this.stdSchedulerFactory = new StdSchedulerFactory();
+    }
     if (this.quartzProps.isPresent() && this.quartzProps.get().size() > 0) {
-      schedulerFactory.initialize(this.quartzProps.get());
+      stdSchedulerFactory.initialize(this.quartzProps.get());

Review Comment:
   NBD, but for an instance variable, the name `schedulerFactory` seems 
clearest to me



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+import org.quartz.*;
+import org.quartz.impl.StdSchedulerFactory;
+
+import javax.inject.Inject;
+
+import org.apache.gobblin.runtime.api.DagActionStore;
+
+
+/**
+ * This class is used to keep track of reminders of pending flow action events 
to execute. A host calls the
+ * {#scheduleReminderJob} on a flow action that it failed to acquire a lease 
on but has not yet completed. The reminder
+ * will fire once the previous lease owner's lease is expected to expire.
+ */
+public class DagProcReminderScheduler {
+  private final Scheduler quartzScheduler;
+  private final DagManagementTaskStreamImpl taskStream;

Review Comment:
   `DagManagement`
   
   (and when you change the type, suggest also to change the member name)



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+import org.quartz.*;
+import org.quartz.impl.StdSchedulerFactory;
+
+import javax.inject.Inject;
+
+import org.apache.gobblin.runtime.api.DagActionStore;
+
+
+/**
+ * This class is used to keep track of reminders of pending flow action events 
to execute. A host calls the
+ * {#scheduleReminderJob} on a flow action that it failed to acquire a lease 
on but has not yet completed. The reminder
+ * will fire once the previous lease owner's lease is expected to expire.
+ */
+public class DagProcReminderScheduler {
+  private final Scheduler quartzScheduler;
+  private final DagManagementTaskStreamImpl taskStream;
+
+  @Inject
+  public DagProcReminderScheduler(StdSchedulerFactory schedulerFactory, 
DagManagementTaskStreamImpl taskStream)
+      throws SchedulerException {
+    // Create a new Scheduler to be used solely for the DagProc reminders
+    this.quartzScheduler = schedulerFactory.getScheduler("DagProcScheduler");
+    this.taskStream = taskStream;
+  }
+
+  /**
+   *  Uses a dagAction & reminder duration in milliseconds to create a 
reminder job that will fire
+   *  `reminderDurationMillis` after the current time
+   * @param dagAction
+   * @param reminderDurationMillis
+   * @throws SchedulerException
+   */
+  public void scheduleReminderJob(DagActionStore.DagAction dagAction, long 
reminderDurationMillis)

Review Comment:
   despite quartz calling it `scheduleJob`, for our purposes, it's just a 
callback, and unrelated to the gobblin jobs gaas launches.
   
   let's maintain abstraction even in our naming, and call it simply 
`scheduleReminder`



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceGuiceModule.java:
##########
@@ -166,6 +168,10 @@ public void configure(Binder binder) {
     binder.bindConstant()
         .annotatedWith(Names.named(InjectionNames.DAG_PROC_ENGINE_ENABLED))
         .to(serviceConfig.isDagProcessingEngineEnabled());
+    // TODO: see if this useful

Review Comment:
   can we remove TODO?



##########
gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerService.java:
##########
@@ -60,20 +61,27 @@ public SchedulerService(Properties props) {
         Optional.of(PropertiesUtils.extractPropertiesWithPrefix(props, 
Optional.of("org.quartz."))));
   }
 
-  @Inject
   public SchedulerService(Config cfg) {
     this(cfg.hasPath(ConfigurationKeys.SCHEDULER_WAIT_FOR_JOB_COMPLETION_KEY) ?
          
cfg.getBoolean(ConfigurationKeys.SCHEDULER_WAIT_FOR_JOB_COMPLETION_KEY) :
          
Boolean.parseBoolean(ConfigurationKeys.DEFAULT_SCHEDULER_WAIT_FOR_JOB_COMPLETION),
          Optional.of(ConfigUtils.configToProperties(cfg, "org.quartz.")));
   }
 
+  @Inject
+  public SchedulerService(Config cfg, StdSchedulerFactory stdSchedulerFactory) 
{
+    this(cfg);
+    this.stdSchedulerFactory = stdSchedulerFactory;
+  }
+
   @Override protected void startUp() throws SchedulerException  {
-    StdSchedulerFactory schedulerFactory = new StdSchedulerFactory();
+    if (this.stdSchedulerFactory == null) {
+      this.stdSchedulerFactory = new StdSchedulerFactory();
+    }

Review Comment:
   a best practice is to have only one "central" ctor per class.  (this is 
actually required in scala.)
   
   simplest way to achieve would be to consolidate all impl in the two-param 
form, by moving over the logic from the one-param ctor.
   
   that hollows out the one-param form simply to forward to the two-param, 
passing `null` as the second arg.
   
   finally, the `null`-handling case in `startUp` could be moved to the 
two-param form



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -230,15 +230,16 @@ public void orchestrate(Spec spec, Properties jobProps, 
long triggerTimestampMil
       Map<String, String> flowMetadata = 
TimingEventUtils.getFlowMetadata(flowSpec);
       String flowExecutionId = 
String.valueOf(FlowUtils.getOrCreateFlowExecutionId(flowSpec));
 
+      // Note we use flowName as jobName for lease arbitration since 
orchestration is at a flow level
       DagActionStore.DagAction flowAction =
-          new DagActionStore.DagAction(flowGroup, flowName, flowExecutionId, 
DagActionStore.FlowActionType.LAUNCH);
+          new DagActionStore.DagAction(flowGroup, flowName, flowExecutionId, 
flowName, DagActionStore.FlowActionType.LAUNCH);

Review Comment:
   are we proceeding w/ this... or did we decide to use `""` to mark the "NO 
jobName" situation?



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MultiActiveLeaseArbiter.java:
##########
@@ -109,8 +118,8 @@ public long getEventTimeMillis() {
    */
   @Data
   class LeasedToAnotherStatus extends LeaseAttemptStatus {
-    private final DagActionStore.DagAction flowAction;
-    private final long minimumLingerDurationMillis;
+    public final DagActionStore.DagAction flowAction;
+    public final long minimumLingerDurationMillis;

Review Comment:
   why not use the getters `@Data` generates?



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/FlowTriggerHandlerTest.java:
##########


Review Comment:
   do you want to rename this class/file?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+import org.quartz.*;
+import org.quartz.impl.StdSchedulerFactory;

Review Comment:
   I see you reaching into the `impl` pkg.  is there something this provides 
that the "non-impl" `org.quartz.Scheduler` interface doesn't contain among its 
publicly-advertised capabilities?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;

Review Comment:
   need license



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts

Review Comment:
   shouldn't it call the backend-agnostic MALA interface (w/o `Mysql` prefix)?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/InstrumentedLeaseArbiterDecorator.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration;
+
+import java.io.IOException;
+
+import com.typesafe.config.Config;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.instrumented.Instrumented;
+import org.apache.gobblin.metrics.ContextAwareCounter;
+import org.apache.gobblin.metrics.ContextAwareMeter;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.metrics.ServiceMetricNames;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/*
+  A generic lease arbitration decorator built upon the {@link 
MysqlMultiActiveLeaseArbiter} which encapsulates common
+  functionality desired by lease arbiter users to track metrics on all lease 
attempts made by the arbiter. The metrics
+  can be used to compare relative performance among arbitration participants.
+ */
+@Slf4j
+public abstract class InstrumentedLeaseArbiterDecorator extends 
MysqlMultiActiveLeaseArbiter {

Review Comment:
   [the Decorator Pattern](https://en.wikipedia.org/wiki/Decorator_pattern) is 
really an alternative to subclassing, so I'd expect it to implement the MALA 
interface rather than derive from the mysql-specific impl.
   
   it is good that "the decorated", `multiActiveLeaseArbiter` member, is 
unconstrained and may be any MALA impl.



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+import org.quartz.*;

Review Comment:
   personally, I don't mind... but I believe coding standard is to avoid 
wildcard imports



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;

Review Comment:
   need license



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/InstrumentedLeaseArbiterDecorator.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration;
+
+import java.io.IOException;
+
+import com.typesafe.config.Config;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.instrumented.Instrumented;
+import org.apache.gobblin.metrics.ContextAwareCounter;
+import org.apache.gobblin.metrics.ContextAwareMeter;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.metrics.ServiceMetricNames;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/*
+  A generic lease arbitration decorator built upon the {@link 
MysqlMultiActiveLeaseArbiter} which encapsulates common
+  functionality desired by lease arbiter users to track metrics on all lease 
attempts made by the arbiter. The metrics
+  can be used to compare relative performance among arbitration participants.
+ */
+@Slf4j
+public abstract class InstrumentedLeaseArbiterDecorator extends 
MysqlMultiActiveLeaseArbiter {
+  protected MultiActiveLeaseArbiter multiActiveLeaseArbiter;
+  protected MetricContext metricContext;
+  private ContextAwareCounter leaseObtainedCount;
+
+  private ContextAwareCounter leasedToAnotherStatusCount;
+
+  private ContextAwareCounter noLongerLeasingStatusCount;
+  private ContextAwareMeter leasesObtainedDueToReminderCount;
+
+  public InstrumentedLeaseArbiterDecorator(Config config, 
MultiActiveLeaseArbiter leaseDeterminationStore,
+      String metricsPrefix) throws IOException {
+    super(config);
+    this.multiActiveLeaseArbiter = leaseDeterminationStore;
+    this.metricContext = Instrumented.getMetricContext(new 
org.apache.gobblin.configuration.State(ConfigUtils.configToProperties(config)),
+        this.getClass());
+    initializeMetrics(metricsPrefix);
+  }
+
+  private void initializeMetrics(String metricsPrefix) {
+    // If a valid metrics prefix is provided then add a delimiter after it
+    if (metricsPrefix != "") {
+      metricsPrefix += ".";
+    }
+    this.leaseObtainedCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASE_OBTAINED_COUNT);
+    this.leasedToAnotherStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASED_TO_ANOTHER_COUNT);
+    this.noLongerLeasingStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_NO_LONGER_LEASING_COUNT);
+    this.leasesObtainedDueToReminderCount = 
this.metricContext.contextAwareMeter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASES_OBTAINED_DUE_TO_REMINDER_COUNT);
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) throws 
IOException {
+
+    MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus =
+        multiActiveLeaseArbiter.tryAcquireLease(flowAction, eventTimeMillis, 
isReminderEvent,
+            skipFlowExecutionIdReplacement);
+    // The flow action contained in the`LeaseAttemptStatus` from the lease 
arbiter contains an updated flow execution
+    // id. From this point onwards, always use the newer version of the flow 
action to easily track the action through
+    // orchestration and execution.
+    if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeaseObtainedStatus) {
+      if (isReminderEvent) {
+        this.leasesObtainedDueToReminderCount.mark();
+      }
+      this.leaseObtainedCount.inc();
+      return leaseAttemptStatus;
+    } else if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+      this.leasedToAnotherStatusCount.inc();
+//        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+      return leaseAttemptStatus;
+    } else if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.NoLongerLeasingStatus) {
+      this.noLongerLeasingStatusCount.inc();
+      log.debug("Received type of leaseAttemptStatus: [{}, eventTimestamp: {}] 
", leaseAttemptStatus.getClass().getName(),

Review Comment:
   why `.debug` and why only on this one?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/InstrumentedLeaseArbiterDecorator.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration;
+
+import java.io.IOException;
+
+import com.typesafe.config.Config;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.instrumented.Instrumented;
+import org.apache.gobblin.metrics.ContextAwareCounter;
+import org.apache.gobblin.metrics.ContextAwareMeter;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.metrics.ServiceMetricNames;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/*
+  A generic lease arbitration decorator built upon the {@link 
MysqlMultiActiveLeaseArbiter} which encapsulates common
+  functionality desired by lease arbiter users to track metrics on all lease 
attempts made by the arbiter. The metrics
+  can be used to compare relative performance among arbitration participants.
+ */
+@Slf4j
+public abstract class InstrumentedLeaseArbiterDecorator extends 
MysqlMultiActiveLeaseArbiter {
+  protected MultiActiveLeaseArbiter multiActiveLeaseArbiter;
+  protected MetricContext metricContext;
+  private ContextAwareCounter leaseObtainedCount;
+
+  private ContextAwareCounter leasedToAnotherStatusCount;
+
+  private ContextAwareCounter noLongerLeasingStatusCount;
+  private ContextAwareMeter leasesObtainedDueToReminderCount;
+
+  public InstrumentedLeaseArbiterDecorator(Config config, 
MultiActiveLeaseArbiter leaseDeterminationStore,
+      String metricsPrefix) throws IOException {
+    super(config);
+    this.multiActiveLeaseArbiter = leaseDeterminationStore;
+    this.metricContext = Instrumented.getMetricContext(new 
org.apache.gobblin.configuration.State(ConfigUtils.configToProperties(config)),
+        this.getClass());
+    initializeMetrics(metricsPrefix);
+  }
+
+  private void initializeMetrics(String metricsPrefix) {
+    // If a valid metrics prefix is provided then add a delimiter after it
+    if (metricsPrefix != "") {
+      metricsPrefix += ".";
+    }
+    this.leaseObtainedCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASE_OBTAINED_COUNT);
+    this.leasedToAnotherStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASED_TO_ANOTHER_COUNT);
+    this.noLongerLeasingStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_NO_LONGER_LEASING_COUNT);
+    this.leasesObtainedDueToReminderCount = 
this.metricContext.contextAwareMeter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASES_OBTAINED_DUE_TO_REMINDER_COUNT);
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) throws 
IOException {
+
+    MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus =
+        multiActiveLeaseArbiter.tryAcquireLease(flowAction, eventTimeMillis, 
isReminderEvent,
+            skipFlowExecutionIdReplacement);
+    // The flow action contained in the`LeaseAttemptStatus` from the lease 
arbiter contains an updated flow execution
+    // id. From this point onwards, always use the newer version of the flow 
action to easily track the action through
+    // orchestration and execution.

Review Comment:
   doesn't seem to belong in this decorator



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/InstrumentedLeaseArbiterDecorator.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.service.modules.orchestration;
+
+import java.io.IOException;
+
+import com.typesafe.config.Config;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.instrumented.Instrumented;
+import org.apache.gobblin.metrics.ContextAwareCounter;
+import org.apache.gobblin.metrics.ContextAwareMeter;
+import org.apache.gobblin.metrics.MetricContext;
+import org.apache.gobblin.metrics.ServiceMetricNames;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+import org.apache.gobblin.util.ConfigUtils;
+
+
+/*
+  A generic lease arbitration decorator built upon the {@link 
MysqlMultiActiveLeaseArbiter} which encapsulates common
+  functionality desired by lease arbiter users to track metrics on all lease 
attempts made by the arbiter. The metrics
+  can be used to compare relative performance among arbitration participants.
+ */
+@Slf4j
+public abstract class InstrumentedLeaseArbiterDecorator extends 
MysqlMultiActiveLeaseArbiter {
+  protected MultiActiveLeaseArbiter multiActiveLeaseArbiter;
+  protected MetricContext metricContext;
+  private ContextAwareCounter leaseObtainedCount;
+
+  private ContextAwareCounter leasedToAnotherStatusCount;
+
+  private ContextAwareCounter noLongerLeasingStatusCount;
+  private ContextAwareMeter leasesObtainedDueToReminderCount;
+
+  public InstrumentedLeaseArbiterDecorator(Config config, 
MultiActiveLeaseArbiter leaseDeterminationStore,
+      String metricsPrefix) throws IOException {
+    super(config);
+    this.multiActiveLeaseArbiter = leaseDeterminationStore;
+    this.metricContext = Instrumented.getMetricContext(new 
org.apache.gobblin.configuration.State(ConfigUtils.configToProperties(config)),
+        this.getClass());
+    initializeMetrics(metricsPrefix);
+  }
+
+  private void initializeMetrics(String metricsPrefix) {
+    // If a valid metrics prefix is provided then add a delimiter after it
+    if (metricsPrefix != "") {
+      metricsPrefix += ".";
+    }
+    this.leaseObtainedCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASE_OBTAINED_COUNT);
+    this.leasedToAnotherStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASED_TO_ANOTHER_COUNT);
+    this.noLongerLeasingStatusCount = 
this.metricContext.contextAwareCounter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_NO_LONGER_LEASING_COUNT);
+    this.leasesObtainedDueToReminderCount = 
this.metricContext.contextAwareMeter(metricsPrefix + 
ServiceMetricNames.FLOW_TRIGGER_HANDLER_LEASES_OBTAINED_DUE_TO_REMINDER_COUNT);
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) throws 
IOException {
+
+    MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus =
+        multiActiveLeaseArbiter.tryAcquireLease(flowAction, eventTimeMillis, 
isReminderEvent,
+            skipFlowExecutionIdReplacement);
+    // The flow action contained in the`LeaseAttemptStatus` from the lease 
arbiter contains an updated flow execution
+    // id. From this point onwards, always use the newer version of the flow 
action to easily track the action through
+    // orchestration and execution.
+    if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeaseObtainedStatus) {
+      if (isReminderEvent) {
+        this.leasesObtainedDueToReminderCount.mark();
+      }
+      this.leaseObtainedCount.inc();
+      return leaseAttemptStatus;
+    } else if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+      this.leasedToAnotherStatusCount.inc();
+//        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);

Review Comment:
   remove



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{

Review Comment:
   as mentioned, a Decorator ought to implement the base interface, not extend 
another impl.
   
   thus rather than an inheritance that delegates each to `super`, apply the 
decorative behaviors by "stacking" the ctor invocations:
   ```
   new DagProcArbiterDecorator(
       new InstrumentedLADecorator(
          new MysqlMALA(...)));
   ```



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";
+
+    @Override
+    public void execute(JobExecutionContext context) throws 
JobExecutionException {
+      // Get properties from the trigger to create a dagAction
+      String flowName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_NAME_KEY);
+      String flowGroup = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_GROUP_KEY);
+      String jobName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.JOB_NAME_KEY);
+      String flowId = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_EXECUTION_ID_KEY);
+      DagActionStore.FlowActionType flowActionType = 
DagActionStore.FlowActionType.valueOf(
+          
context.getTrigger().getJobDataMap().getString(FLOW_ACTION_TYPE_KEY));
+       DagManagementTaskStreamImpl dagManagementTaskStream =

Review Comment:
   any given client should use one interface, but NOT both, either 
`DagManagement` OR `DagTaskStream`.  since the only method used here seems to 
be `addDagAction`, the type should be `DagManagement`.
   
   (changing here will open the door to simplifying several other type decls on 
the way to tunneling it into here.)



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/FlowTriggerDecorator.java:
##########
@@ -116,54 +100,41 @@ public FlowTriggerHandler(Config config, 
Optional<MultiActiveLeaseArbiter> lease
    * @throws IOException
    */
   public void handleTriggerEvent(Properties jobProps, DagActionStore.DagAction 
flowAction, long eventTimeMillis,
-      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) throws 
IOException {
-    if (multiActiveLeaseArbiter.isPresent()) {
-      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
multiActiveLeaseArbiter.get().tryAcquireLease(
-          flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
-      // The flow action contained in the`LeaseAttemptStatus` from the lease 
arbiter contains an updated flow execution
-      // id. From this point onwards, always use the newer version of the flow 
action to easily track the action through
-      // orchestration and execution.
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement)
+      throws IOException {
+
+    try {
+      LeaseAttemptStatus leaseAttemptStatus = null;
+      leaseAttemptStatus =

Review Comment:
   why first initialize to `null`?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcReminderScheduler.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+import org.quartz.*;
+import org.quartz.impl.StdSchedulerFactory;
+
+import javax.inject.Inject;
+
+import org.apache.gobblin.runtime.api.DagActionStore;
+
+
+/**
+ * This class is used to keep track of reminders of pending flow action events 
to execute. A host calls the
+ * {#scheduleReminderJob} on a flow action that it failed to acquire a lease 
on but has not yet completed. The reminder
+ * will fire once the previous lease owner's lease is expected to expire.
+ */
+public class DagProcReminderScheduler {

Review Comment:
   nit: this doesn't schedule reminders on `DagProc`s, but on `DagAction`s, 
hence `DagActionReminderScheduler` or `DagActionReminderService`



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/FlowTriggerDecorator.java:
##########
@@ -116,54 +100,41 @@ public FlowTriggerHandler(Config config, 
Optional<MultiActiveLeaseArbiter> lease
    * @throws IOException
    */
   public void handleTriggerEvent(Properties jobProps, DagActionStore.DagAction 
flowAction, long eventTimeMillis,
-      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) throws 
IOException {
-    if (multiActiveLeaseArbiter.isPresent()) {
-      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
multiActiveLeaseArbiter.get().tryAcquireLease(
-          flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
-      // The flow action contained in the`LeaseAttemptStatus` from the lease 
arbiter contains an updated flow execution
-      // id. From this point onwards, always use the newer version of the flow 
action to easily track the action through
-      // orchestration and execution.
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement)
+      throws IOException {
+
+    try {
+      LeaseAttemptStatus leaseAttemptStatus = null;
+      leaseAttemptStatus =
+          this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
       if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeaseObtainedStatus) {
-        if (isReminderEvent) {
-          this.leasesObtainedDueToReminderCount.mark();
+          MultiActiveLeaseArbiter.LeaseObtainedStatus leaseObtainedStatus = 
(MultiActiveLeaseArbiter.LeaseObtainedStatus)
+              leaseAttemptStatus;
+          if (persistFlowAction(leaseObtainedStatus)) {

Review Comment:
   I believe at one point, originally, this terminology of "flow action" made 
sense to me, but as we've gone deeper to define a bevy of new actions pretty 
exclusively around the name `DagAction`, I'm left to wonder:
   * what is a "flow action"?
   * how is it different from a DagAction?
   * could the two concepts be consolidated into one, to also unify the 
naming/terminology?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;

Review Comment:
   as discussed elsewhere, I believe we ought to set the reminder EXCEPT when 
`NoLongerLeasing`.  that defensively guards against "leaking" the action when 
this host obtains the lease, but fails in executing the action



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/FlowTriggerDecorator.java:
##########
@@ -64,43 +62,29 @@
  * in failure cases.
  */
 @Slf4j
-public class FlowTriggerHandler {
+public class FlowTriggerDecorator extends InstrumentedLeaseArbiterDecorator {

Review Comment:
   this naming suggests a type named `FlowTrigger` is to be decorated... 
although w/o specifying how.  (cf. `SelfHealingFlowTriggerDecorator`, 
`BufferedFlowTriggerDecorator`, etc.)



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/FlowTriggerDecorator.java:
##########
@@ -64,43 +62,29 @@
  * in failure cases.
  */
 @Slf4j
-public class FlowTriggerHandler {
+public class FlowTriggerDecorator extends InstrumentedLeaseArbiterDecorator {
+  private Optional<DagActionStore> dagActionStore;
   private final int schedulerMaxBackoffMillis;
   private static Random random = new Random();
-  protected Optional<MultiActiveLeaseArbiter> multiActiveLeaseArbiter;
   protected SchedulerService schedulerService;
-  protected Optional<DagActionStore> dagActionStore;
-  private MetricContext metricContext;
   private ContextAwareMeter numFlowsSubmitted;
-
-  private ContextAwareCounter leaseObtainedCount;
-
-  private ContextAwareCounter leasedToAnotherStatusCount;
-
-  private ContextAwareCounter noLongerLeasingStatusCount;
   private ContextAwareCounter jobDoesNotExistInSchedulerCount;
   private ContextAwareCounter failedToSetEventReminderCount;
-  private ContextAwareMeter leasesObtainedDueToReminderCount;
   private ContextAwareMeter failedToRecordLeaseSuccessCount;
   private ContextAwareMeter recordedLeaseSuccessCount;
 
   @Inject
-  public FlowTriggerHandler(Config config, Optional<MultiActiveLeaseArbiter> 
leaseDeterminationStore,
-      SchedulerService schedulerService, Optional<DagActionStore> 
dagActionStore) {
+  public FlowTriggerDecorator(Config config, Optional<MultiActiveLeaseArbiter> 
leaseDeterminationStore,
+      SchedulerService schedulerService, Optional<DagActionStore> 
dagActionStore)
+      throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(FlowTriggerDecorator.class));

Review Comment:
   `String.valueOf(X.class)` is non-canonical.
   
   prefer: `X.class.getName()` or `X.class.getSimpleName()`



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";

Review Comment:
   let's append the `_KEY` suffix on this one too



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";
+
+    @Override
+    public void execute(JobExecutionContext context) throws 
JobExecutionException {
+      // Get properties from the trigger to create a dagAction
+      String flowName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_NAME_KEY);
+      String flowGroup = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_GROUP_KEY);
+      String jobName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.JOB_NAME_KEY);
+      String flowId = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_EXECUTION_ID_KEY);

Review Comment:
   the repetitive `context.getTrigger().getJobDataMap().` adds clutter.  could 
we just define a locally-scoped `dataMap` at the top?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";
+
+    @Override
+    public void execute(JobExecutionContext context) throws 
JobExecutionException {
+      // Get properties from the trigger to create a dagAction
+      String flowName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_NAME_KEY);
+      String flowGroup = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_GROUP_KEY);
+      String jobName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.JOB_NAME_KEY);
+      String flowId = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_EXECUTION_ID_KEY);
+      DagActionStore.FlowActionType flowActionType = 
DagActionStore.FlowActionType.valueOf(
+          
context.getTrigger().getJobDataMap().getString(FLOW_ACTION_TYPE_KEY));
+       DagManagementTaskStreamImpl dagManagementTaskStream =
+           (DagManagementTaskStreamImpl) 
context.getTrigger().getJobDataMap().get(DAG_TASK_STREAM);
+
+      // TODO: add meaningful log statement
+      log.info("Reminder for job " + jobName + " in flow " + flowName + " (" + 
flowGroup + ") with ID " + flowId);
+
+      DagActionStore.DagAction dagAction = new 
DagActionStore.DagAction(flowGroup, flowName, flowId, jobName, flowActionType);
+
+       dagManagementTaskStream.addDagAction(dagAction);
+    }
+  }
+
+  public static String createDagActionReminderKey(DagActionStore.DagAction 
dagAction) {
+    return createDagActionReminderKey(dagAction.getFlowName(), 
dagAction.getFlowGroup(), dagAction.getJobName(),
+        dagAction.getFlowExecutionId(), dagAction.getFlowActionType());
+  }
+
+  public static String createDagActionReminderKey(String flowName, String 
flowGroup, String jobName, String flowId,
+      DagActionStore.FlowActionType flowActionType) {
+    return String.format("%s.%s.%s.%s.%s", flowName, flowGroup, jobName, 
flowId, flowActionType);
+  }
+
+  public static JobDetail createReminderJobDetail(DagManagementTaskStreamImpl 
taskStream, DagActionStore.DagAction dagAction) {
+    JobDataMap dataMap = new JobDataMap();
+    dataMap.put(ReminderJob.DAG_TASK_STREAM, taskStream);

Review Comment:
   what does it mean to stuff this into a data map?  if it involves 
serialization, I'm not in favor, because the `DagMgmtTaskStreamImpl` could have 
lengthy internal queues.
   
   given there's only one DMTSImpl per JVM, do we really need it to ride along 
within the reminder?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";
+
+    @Override
+    public void execute(JobExecutionContext context) throws 
JobExecutionException {
+      // Get properties from the trigger to create a dagAction
+      String flowName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_NAME_KEY);
+      String flowGroup = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_GROUP_KEY);
+      String jobName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.JOB_NAME_KEY);
+      String flowId = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_EXECUTION_ID_KEY);
+      DagActionStore.FlowActionType flowActionType = 
DagActionStore.FlowActionType.valueOf(
+          
context.getTrigger().getJobDataMap().getString(FLOW_ACTION_TYPE_KEY));
+       DagManagementTaskStreamImpl dagManagementTaskStream =
+           (DagManagementTaskStreamImpl) 
context.getTrigger().getJobDataMap().get(DAG_TASK_STREAM);
+
+      // TODO: add meaningful log statement
+      log.info("Reminder for job " + jobName + " in flow " + flowName + " (" + 
flowGroup + ") with ID " + flowId);
+
+      DagActionStore.DagAction dagAction = new 
DagActionStore.DagAction(flowGroup, flowName, flowId, jobName, flowActionType);
+
+       dagManagementTaskStream.addDagAction(dagAction);
+    }
+  }
+
+  public static String createDagActionReminderKey(DagActionStore.DagAction 
dagAction) {
+    return createDagActionReminderKey(dagAction.getFlowName(), 
dagAction.getFlowGroup(), dagAction.getJobName(),
+        dagAction.getFlowExecutionId(), dagAction.getFlowActionType());
+  }
+
+  public static String createDagActionReminderKey(String flowName, String 
flowGroup, String jobName, String flowId,
+      DagActionStore.FlowActionType flowActionType) {
+    return String.format("%s.%s.%s.%s.%s", flowName, flowGroup, jobName, 
flowId, flowActionType);

Review Comment:
   logically, in increasing order of specificity, the ordering would seem:
   flowGroup / flowName / flowExecId / jobName
   
   AFAIR this ordering is used in other logging, to name jobs, and to create 
temp/staging dirs.  could we adopt that format here?



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagProcArbiterDecorator.java:
##########
@@ -0,0 +1,140 @@
+package org.apache.gobblin.service.modules.orchestration;
+
+
+import java.io.IOException;
+import java.util.Date;
+
+import org.quartz.Job;
+import org.quartz.JobBuilder;
+import org.quartz.JobDataMap;
+import org.quartz.JobDetail;
+import org.quartz.JobExecutionContext;
+import org.quartz.JobExecutionException;
+import org.quartz.SchedulerException;
+import org.quartz.Trigger;
+import org.quartz.TriggerBuilder;
+
+import com.google.common.base.Optional;
+import com.typesafe.config.Config;
+
+import javax.inject.Inject;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.api.DagActionStore;
+import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
+import org.apache.gobblin.runtime.api.MysqlMultiActiveLeaseArbiter;
+
+
+/**
+ * Decorator used to coordinate multiple hosts with execution components 
enabled to respond to flow action events. It
+ * uses the {@link MultiActiveLeaseArbiter} to determine a single lease owner 
at a given event time for a flow action
+ * event. After acquiring the lease, the host can pursue executing the action. 
Once it has completed this action, it
+ * marks the lease as completed by calling the
+ * {@link 
MysqlMultiActiveLeaseArbiter#recordLeaseSuccess(MultiActiveLeaseArbiter.LeaseObtainedStatus)}
 method. Hosts
+ * that fail to acquire a lease will use the {@link DagProcReminderScheduler} 
to set a reminder for the flow action
+ * event to check back in on the previous lease owner's completion status.
+ */
+public class DagProcArbiterDecorator extends InstrumentedLeaseArbiterDecorator 
{
+  private final DagProcReminderScheduler dagProcReminderScheduler;
+
+  @Inject
+  public DagProcArbiterDecorator(Config config, 
Optional<MultiActiveLeaseArbiter> leaseDeterminationStore,
+      DagProcReminderScheduler dagProcReminderScheduler) throws IOException {
+    super(config, leaseDeterminationStore.get(), 
String.valueOf(DagProcArbiterDecorator.class));
+    this.dagProcReminderScheduler = dagProcReminderScheduler;
+  }
+
+  /**
+   * This method is used by the multi-active scheduler and multi-active 
execution classes (DagTaskStream) to attempt a
+   * lease for a particular job event and return the status of the attempt.
+   * @param flowAction
+   * @param eventTimeMillis
+   * @param isReminderEvent
+   * @param skipFlowExecutionIdReplacement
+   * @return
+   */
+  @Override
+  public MultiActiveLeaseArbiter.LeaseAttemptStatus 
tryAcquireLease(DagActionStore.DagAction flowAction, long eventTimeMillis,
+      boolean isReminderEvent, boolean skipFlowExecutionIdReplacement) {
+    try {
+      MultiActiveLeaseArbiter.LeaseAttemptStatus leaseAttemptStatus = 
this.tryAcquireLease(flowAction, eventTimeMillis, isReminderEvent, 
skipFlowExecutionIdReplacement);
+      if (leaseAttemptStatus instanceof 
MultiActiveLeaseArbiter.LeasedToAnotherStatus) {
+        
scheduleReminderForEvent((MultiActiveLeaseArbiter.LeasedToAnotherStatus) 
leaseAttemptStatus, eventTimeMillis);
+        return leaseAttemptStatus;
+      }
+    /* Otherwise leaseAttemptStatus instanceof {@link 
MultiActiveLeaseArbiter.LeaseObtainedStatus/NoLongerLeasingStatus}
+      and no need to do anything
+     */
+      return leaseAttemptStatus;
+    } catch (SchedulerException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  protected void 
scheduleReminderForEvent(MultiActiveLeaseArbiter.LeasedToAnotherStatus 
leaseStatus, long triggerEventTimeMillis)
+      throws SchedulerException {
+    // TODO: determine which ts to use
+    dagProcReminderScheduler.scheduleReminderJob(leaseStatus.getFlowAction(), 
leaseStatus.getEventTimeMillis());
+
+  }
+
+  @Slf4j
+  public class ReminderJob implements Job {
+    public static final String FLOW_ACTION_TYPE_KEY = "flow.actionType";
+    public static final String DAG_TASK_STREAM = "dag.taskStream";
+
+    @Override
+    public void execute(JobExecutionContext context) throws 
JobExecutionException {
+      // Get properties from the trigger to create a dagAction
+      String flowName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_NAME_KEY);
+      String flowGroup = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_GROUP_KEY);
+      String jobName = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.JOB_NAME_KEY);
+      String flowId = 
context.getTrigger().getJobDataMap().getString(ConfigurationKeys.FLOW_EXECUTION_ID_KEY);
+      DagActionStore.FlowActionType flowActionType = 
DagActionStore.FlowActionType.valueOf(
+          
context.getTrigger().getJobDataMap().getString(FLOW_ACTION_TYPE_KEY));
+       DagManagementTaskStreamImpl dagManagementTaskStream =
+           (DagManagementTaskStreamImpl) 
context.getTrigger().getJobDataMap().get(DAG_TASK_STREAM);
+
+      // TODO: add meaningful log statement
+      log.info("Reminder for job " + jobName + " in flow " + flowName + " (" + 
flowGroup + ") with ID " + flowId);

Review Comment:
   agree with TODO: this is way too conversational.  suggest to appropriate the 
syntax of the REST API - e.g.
   ```
   (flowGroup:myflowgroup,flowName:myflowname,flowExecutionId:12345)
   ```
   and maybe throw `jobName` in as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to