[
https://issues.apache.org/jira/browse/GOBBLIN-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271436#comment-16271436
]
Abhishek Tiwari commented on GOBBLIN-311:
-----------------------------------------
[~jbaranick] We probably should add some sort of cleanup at bootup time. The
cluster could easily crash as well in case of standalone cluster, and thus have
some remanent jobs persisted that would be tried again.
> Gobblin AWS runs old jobs when cluster is restarted.
> ----------------------------------------------------
>
> Key: GOBBLIN-311
> URL: https://issues.apache.org/jira/browse/GOBBLIN-311
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Joel Baranick
>
> On startup of my cluster, old jobs are still attempted. @htran1 said that
> they should be cleaned up in Standalone mode, but that does not seem
> compatible with running under AWS:
> [http://gobblin.readthedocs.io/en/latest/user-guide/Gobblin-Deployment/#standalone-architecture]
> Also, if I enabled Standalone mode, then
> {{GobblinClusterManager.sendShutdownRequest()}} won't be called.
> Additionally, when enabling Standalone mode, GobblinClusterManager will call
> the following code, which doesn't seem right if I'm running under AWS:
> {code:java}
> // In AWS / Yarn mode, the cluster Launcher takes care of setting up Helix
> cluster
> /// .. but for Standalone mode, we go via this main() method, so setup the
> cluster here
> if (isStandaloneClusterManager) {
> // Create Helix cluster and connect to it
> String zkConnectionString =
> config.getString(GobblinClusterConfigurationKeys.ZK_CONNECTION_STRING_KEY);
> String helixClusterName =
> config.getString(GobblinClusterConfigurationKeys.HELIX_CLUSTER_NAME_KEY);
> HelixUtils.createGobblinHelixCluster(zkConnectionString,
> helixClusterName, false);
> LOGGER.info("Created Helix cluster " + helixClusterName);
> }
> {code}
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
