[
https://issues.apache.org/jira/browse/GOBBLIN-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudarshan Vasudevan updated GOBBLIN-687:
----------------------------------------
Summary: Pass TopologySpec map to DagManager to allow reuse of
SpecExecutors during DAG deserialization (was: Pass TopologySpec map to
DagManager )
> Pass TopologySpec map to DagManager to allow reuse of SpecExecutors during
> DAG deserialization
> ----------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-687
> URL: https://issues.apache.org/jira/browse/GOBBLIN-687
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Affects Versions: 0.15.0
> Reporter: Sudarshan Vasudevan
> Assignee: Abhishek Tiwari
> Priority: Major
> Fix For: 0.15.0
>
>
> DagManager maintains state of all currently executing DAGs, by serializing
> each DAG on compilation and persisting it to a durable store. The serialized
> DAG includes Job config as well as the SpecExecutor config for each job in
> the DAG. This is done to correctly resume execution of DAGs in case of
> service restarts or leadership change.
> Currently, on service restart/leadership change, the new master de-serializes
> SpecExecutor config and creates a SpecExecutor instance for each job in the
> DAG. If the number of DAGs is large, this can result in many connections to
> the underlying executor instance. The proposed fix allows the DagManager to
> re-use the SpecExecutor instances created by the TopologySpecFactory when it
> deserializes a DAG.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
