phet commented on code in PR #3858:
URL: https://github.com/apache/gobblin/pull/3858#discussion_r1518414282
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManagementTaskStreamImpl.java:
##########
@@ -36,16 +35,14 @@
import org.apache.gobblin.metrics.event.EventSubmitter;
import org.apache.gobblin.runtime.api.DagActionStore;
import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
-import org.apache.gobblin.service.modules.orchestration.proc.DagProc;
import org.apache.gobblin.service.modules.orchestration.task.DagTask;
import org.apache.gobblin.service.modules.orchestration.task.LaunchDagTask;
import org.apache.gobblin.util.ConfigUtils;
/**
- * DagManagementTaskStreamImpl has these functionalities :
- * a) interact with {@link DagManagementStateStore} to update/retrieve dags,
checkpoint
- * b) add {@link org.apache.gobblin.runtime.api.DagActionStore.DagAction} to
the {@link DagTaskStream}
+ * DagManagementTaskStreamImpl implements {@link DagManagement} and {@link
DagTaskStream}. It accepts
+ * {@link org.apache.gobblin.runtime.api.DagActionStore.DagAction}s and
iteratively provides {@link DagTask}.
Review Comment:
I love reading this comment: beautifully cohesive abstraction!
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/DagProcessingEngineTest.java:
##########
@@ -102,34 +102,43 @@ public boolean hasNext() {
}
@Override
- public synchronized DagTask next() {
+ public synchronized DagTask next() throws NoSuchElementException {
i++;
- if (i <= MAX_NUM_OF_TASKS) {
- return new MockedDagTask(new DagActionStore.DagAction("fg-" + i, "fn-"
+ i, "1234" + i, DagActionStore.FlowActionType.LAUNCH), null);
+ if (i > MAX_NUM_OF_TASKS) {
+ throw new RuntimeException("Simulating an exception to stop the
thread!");
Review Comment:
technically this should be thrown by `MockedDagProc.act`... so a param to
it's ctor (tunneled through the `MockedDagTask::host`)
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/LaunchDagTask.java:
##########
@@ -20,21 +20,18 @@
import org.apache.gobblin.runtime.api.DagActionStore;
import org.apache.gobblin.runtime.api.MultiActiveLeaseArbiter;
import org.apache.gobblin.service.modules.orchestration.DagTaskVisitor;
-import org.apache.gobblin.service.modules.orchestration.proc.LaunchDagProc;
/**
* A {@link DagTask} responsible to handle launch tasks.
*/
-public class LaunchDagTask extends DagTask<LaunchDagProc> {
+public class LaunchDagTask<T> extends DagTask {
public LaunchDagTask(DagActionStore.DagAction dagAction,
MultiActiveLeaseArbiter.LeaseObtainedStatus leaseObtainedStatus) {
super(dagAction, leaseObtainedStatus);
}
-
- @Override
- public LaunchDagProc host(DagTaskVisitor<LaunchDagProc> visitor) {
+ public <T> T host(DagTaskVisitor<T> visitor) {
Review Comment:
not necessary: having a generic method is enough, no need for class-level
too (whose `T` is actually overshadowed by the method's `T`)
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/DagProcessingEngineTest.java:
##########
@@ -150,12 +162,19 @@ protected void commit(DagManagementStateStore
dagManagementStateStore, Void resu
}
}
- // This tests verifies that
+ // This tests verifies that all the dag tasks entered to the dag task stream
are retrieved by dag proc engine threads
@Test
public void dagProcessingTest() throws InterruptedException,
TimeoutException {
+ // there are MAX_NUM_OF_TASKS dag tasks returned and then each thread
additionally call (infinitely) once to wait
+ // in this unit tests, it does not infinitely wait though, because the
mocked task stream throws an exception on
+ // (MAX_NUM_OF_TASKS + 1) th call
int expectedNumOfInvocations = MockedDagTaskStream.MAX_NUM_OF_TASKS +
ServiceConfigKeys.DEFAULT_NUM_DAG_PROC_THREADS;
+ int expectedExceptions = MockedDagTaskStream.MAX_NUM_OF_TASKS /
MockedDagTaskStream.FAILING_DAGS_FREQUENCY;
Review Comment:
this will change back to `MAX_TASKS + numDPEThreads` when the exception is
thrown by `DagProc::act`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]