phet commented on code in PR #3858:
URL: https://github.com/apache/gobblin/pull/3858#discussion_r1518414282


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManagementTaskStreamImpl.java:
##########
@@ -36,16 +35,14 @@
 import org.apache.gobblin.metrics.event.EventSubmitter;
 import org.apache.gobblin.runtime.api.DagActionStore;
 import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
-import org.apache.gobblin.service.modules.orchestration.proc.DagProc;
 import org.apache.gobblin.service.modules.orchestration.task.DagTask;
 import org.apache.gobblin.service.modules.orchestration.task.LaunchDagTask;
 import org.apache.gobblin.util.ConfigUtils;
 
 
 /**
- * DagManagementTaskStreamImpl has these functionalities :
- * a) interact with {@link DagManagementStateStore} to update/retrieve dags, 
checkpoint
- * b) add {@link org.apache.gobblin.runtime.api.DagActionStore.DagAction} to 
the {@link DagTaskStream}
+ * DagManagementTaskStreamImpl implements {@link DagManagement} and {@link 
DagTaskStream}. It accepts
+ * {@link org.apache.gobblin.runtime.api.DagActionStore.DagAction}s and 
iteratively provides {@link DagTask}.

Review Comment:
   I love reading this comment: beautifully cohesive abstraction!



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/DagProcessingEngineTest.java:
##########
@@ -102,34 +102,43 @@ public boolean hasNext() {
     }
 
     @Override
-    public synchronized DagTask next() {
+    public synchronized DagTask next() throws NoSuchElementException {
       i++;
-      if (i <= MAX_NUM_OF_TASKS) {
-        return new MockedDagTask(new DagActionStore.DagAction("fg-" + i, "fn-" 
+ i, "1234" + i, DagActionStore.FlowActionType.LAUNCH), null);
+      if (i > MAX_NUM_OF_TASKS) {
+        throw new RuntimeException("Simulating an exception to stop the 
thread!");

Review Comment:
   technically this should be thrown by `MockedDagProc.act`... so a param to 
it's ctor (tunneled through the `MockedDagTask::host`)



##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/LaunchDagTask.java:
##########
@@ -20,21 +20,18 @@
 import org.apache.gobblin.runtime.api.DagActionStore;
 import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
 import org.apache.gobblin.service.modules.orchestration.DagTaskVisitor;
-import org.apache.gobblin.service.modules.orchestration.proc.LaunchDagProc;
 
 
 /**
  * A {@link DagTask} responsible to handle launch tasks.
  */
 
-public class LaunchDagTask extends DagTask<LaunchDagProc> {
+public class LaunchDagTask<T> extends DagTask {
   public LaunchDagTask(DagActionStore.DagAction dagAction, 
MultiActiveLeaseArbiter.LeaseObtainedStatus leaseObtainedStatus) {
     super(dagAction, leaseObtainedStatus);
   }
 
-
-  @Override
-  public LaunchDagProc host(DagTaskVisitor<LaunchDagProc> visitor) {
+  public <T> T host(DagTaskVisitor<T> visitor) {

Review Comment:
   not necessary: having a generic method is enough, no need for class-level 
too (whose `T` is actually overshadowed by the method's `T`)



##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/DagProcessingEngineTest.java:
##########
@@ -150,12 +162,19 @@ protected void commit(DagManagementStateStore 
dagManagementStateStore, Void resu
     }
   }
 
-  // This tests verifies that
+  // This tests verifies that all the dag tasks entered to the dag task stream 
are retrieved by dag proc engine threads
   @Test
   public void dagProcessingTest() throws InterruptedException, 
TimeoutException {
+    // there are MAX_NUM_OF_TASKS dag tasks returned and then each thread 
additionally call (infinitely) once to wait
+    // in this unit tests, it does not infinitely wait though, because the 
mocked task stream throws an exception on
+    // (MAX_NUM_OF_TASKS + 1) th call
     int expectedNumOfInvocations = MockedDagTaskStream.MAX_NUM_OF_TASKS + 
ServiceConfigKeys.DEFAULT_NUM_DAG_PROC_THREADS;
+    int expectedExceptions = MockedDagTaskStream.MAX_NUM_OF_TASKS / 
MockedDagTaskStream.FAILING_DAGS_FREQUENCY;

Review Comment:
   this will change back to `MAX_TASKS + numDPEThreads` when the exception is 
thrown by `DagProc::act`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to