autumnust commented on a change in pull request #2674: [GOBBLIN-808] implement
azkaban flow cancel when dag manager is enabled
URL: https://github.com/apache/incubator-gobblin/pull/2674#discussion_r302785048
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -187,12 +230,37 @@ protected void startUp() {
synchronized void offer(Dag<JobExecutionPlan> dag) throws IOException {
//Persist the dag
this.dagStateStore.writeCheckpoint(dag);
- //Add it to the queue of dags
- if (!this.queue.offer(dag)) {
+ long flowExecutionId = DagManagerUtils.getFlowExecId(dag);
+ int queueId = (int) (flowExecutionId % this.numThreads);
+ // Add the dag to the specific queue determined by flowExecutionId
+ // Flow cancellation request has to be forwarded to the same
DagManagerThread where the
+ // flow create request was forwarded. This is because Azkaban Exec Id is
stored in the DagNode of the
+ // specific DagManagerThread queue
+ if (!this.queue[queueId].offer(dag)) {
throw new IOException("Could not add dag" +
DagManagerUtils.generateDagId(dag) + "to queue");
}
}
+ /**
+ * Method to submit a {@link URI} for cancellation requsts to the {@link
DagManager}.
+ * The {@link DagManager} adds the dag to the {@link BlockingQueue} to be
picked up by one of the {@link DagManagerThread}s.
+ */
+ synchronized public void offer(URI uri) throws IOException {
Review comment:
Can we rename this method to differentiate with `offer` method above?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
