Joel Baranick created GOBBLIN-311: ------------------------------------- Summary: Gobblin AWS runs old jobs when cluster is restarted. Key: GOBBLIN-311 URL: https://issues.apache.org/jira/browse/GOBBLIN-311 Project: Apache Gobblin Issue Type: Bug Reporter: Joel Baranick Assignee: Hung Tran
On startup of my cluster, old jobs are still attempted. @htran1 said that they should be cleaned up in Standalone mode, but that does not seem compatible with running under AWS: [http://gobblin.readthedocs.io/en/latest/user-guide/Gobblin-Deployment/#standalone-architecture] Also, if I enabled Standalone mode, then {{GobblinClusterManager.sendShutdownRequest()}} won't be called. Additionally, when enabling Standalone mode, GobblinClusterManager will call the following code, which doesn't seem right if I'm running under AWS: {code:java} // In AWS / Yarn mode, the cluster Launcher takes care of setting up Helix cluster /// .. but for Standalone mode, we go via this main() method, so setup the cluster here if (isStandaloneClusterManager) { // Create Helix cluster and connect to it String zkConnectionString = config.getString(GobblinClusterConfigurationKeys.ZK_CONNECTION_STRING_KEY); String helixClusterName = config.getString(GobblinClusterConfigurationKeys.HELIX_CLUSTER_NAME_KEY); HelixUtils.createGobblinHelixCluster(zkConnectionString, helixClusterName, false); LOGGER.info("Created Helix cluster " + helixClusterName); } {code} Thoughts? -- This message was sent by Atlassian JIRA (v6.4.14#64029)