Joel Baranick created GOBBLIN-311:
-------------------------------------
Summary: Gobblin AWS runs old jobs when cluster is restarted.
Key: GOBBLIN-311
URL: https://issues.apache.org/jira/browse/GOBBLIN-311
Project: Apache Gobblin
Issue Type: Bug
Reporter: Joel Baranick
Assignee: Hung Tran
On startup of my cluster, old jobs are still attempted. @htran1 said that they
should be cleaned up in Standalone mode, but that does not seem compatible with
running under AWS:
[http://gobblin.readthedocs.io/en/latest/user-guide/Gobblin-Deployment/#standalone-architecture]
Also, if I enabled Standalone mode, then
{{GobblinClusterManager.sendShutdownRequest()}} won't be called. Additionally,
when enabling Standalone mode, GobblinClusterManager will call the following
code, which doesn't seem right if I'm running under AWS:
{code:java}
// In AWS / Yarn mode, the cluster Launcher takes care of setting up Helix
cluster
/// .. but for Standalone mode, we go via this main() method, so setup the
cluster here
if (isStandaloneClusterManager) {
// Create Helix cluster and connect to it
String zkConnectionString =
config.getString(GobblinClusterConfigurationKeys.ZK_CONNECTION_STRING_KEY);
String helixClusterName =
config.getString(GobblinClusterConfigurationKeys.HELIX_CLUSTER_NAME_KEY);
HelixUtils.createGobblinHelixCluster(zkConnectionString, helixClusterName,
false);
LOGGER.info("Created Helix cluster " + helixClusterName);
}
{code}
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
