[
https://issues.apache.org/jira/browse/GOBBLIN-687?focusedWorklogId=201514&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201514
]
ASF GitHub Bot logged work on GOBBLIN-687:
------------------------------------------
Author: ASF GitHub Bot
Created on: 20/Feb/19 18:45
Start Date: 20/Feb/19 18:45
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2559: GOBBLIN-687:
Pass TopologySpec map to DagManager to allow reuse of Sp…
URL: https://github.com/apache/incubator-gobblin/pull/2559#discussion_r258623894
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/core/GobblinServiceManager.java
##########
@@ -218,6 +211,12 @@ public GobblinServiceManager(String serviceName, String
serviceId, Config config
this.helixManager = Optional.absent();
}
+ // Initialize DagManager
Review comment:
Thanks for noticing this! The DagManager initialization does not depend on
whether FlowCatalog is enabled or not. We have a separate config to
enable/disable DagManager. I can't recall why it was inside if
(isFlowCatalogEnabled). Yes, the move outside the if block is intentional.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 201514)
Time Spent: 1h 10m (was: 1h)
> Pass TopologySpec map to DagManager to allow reuse of SpecExecutors during
> DAG deserialization
> ----------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-687
> URL: https://issues.apache.org/jira/browse/GOBBLIN-687
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Affects Versions: 0.15.0
> Reporter: Sudarshan Vasudevan
> Assignee: Abhishek Tiwari
> Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> DagManager maintains state of all currently executing DAGs, by serializing
> each DAG on compilation and persisting it to a durable store. The serialized
> DAG includes Job config as well as the SpecExecutor config for each job in
> the DAG. This is done to correctly resume execution of DAGs in case of
> service restarts or leadership change.
> Currently, on service restart/leadership change, the new master de-serializes
> SpecExecutor config and creates a SpecExecutor instance for each job in the
> DAG. If the number of DAGs is large, this can result in many connections to
> the underlying executor instance. The proposed fix allows the DagManager to
> re-use the SpecExecutor instances created by the TopologySpecFactory when it
> deserializes a DAG.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
