Kuai Yu created GOBBLIN-480:
-------------------------------
Summary: Allow job distribution cluster to be separated from
cluster manager cluster
Key: GOBBLIN-480
URL: https://issues.apache.org/jira/browse/GOBBLIN-480
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Kuai Yu
Assignee: Kuai Yu
Today GobblinClusterManager leverages single Helix cluster responsible for both
job distribution and cluster manager HA. This all-in-one mode cannot works with
Helix super controller, because GobblinClusterManager will create its own
dedicated controller for HA handling, which is internal to Gobblin framework.
This architect works fine but gradually we find it's hard to monitor Helix
behavior and debug Helix related issues due to the lack of Helix task framework
metrics, which is enabled for free, but only available when using a dedicated
controllers under Helix super controller's supervision.
To allow the migration, we separated existing cluster into two clusters:
1. Our existing cluster will remain the same, called "job distribution
cluster". In unit test or local deployment mode, we will create a dedicated
controller for this cluster. In production mode, we assume Helix will provide
this dedicated controller for us.
2. A new cluster will be created, called 'manager cluster', which is
responsible for cluster manager leadership change. This will leadership change
callback just like we did earlier in all-in-one mode.
Two cluster mode can be turned on/off by user configuration. Similarly to
whether a controller for job distribution should be created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
