I have a use case where I want to start up some services in a particular
order. I am using Zookeeper to store cluster-wide properties that need
to be set before dependent services start. (A shared dbms that needs
some specific records set before worker nodes start presents a similar
situation, for example: extending services/hbase to use MySQL for
metadata.) What is the right way to do this in Whirr? My current ideas
are numbered below.
Looking at ClusterActionHandlerSupport (action callbacks) and:
https://cwiki.apache.org/confluence/display/WHIRR/Implementing+a+New+Service
which states that phases of a Whirr run are essentially cluster-wide
synchronization barriers (bootstrap, configuration, and I assume also
start, stop, cleanup, destroy), but there is no way to specify order
within a phase; I see that:
1. There is a way to perform an action afterConfiguration() which
occurs prior to any calls to beforeStart(), as long as a kind of daemon
only occurs once per cluster (like a master node for HBase) so these
dependent actions would only happen once. This works okay for the
situation where one MySQL could be started up, a new user and db is
created, and then the beforeStart() for mysql (if mysql was a Whirr
service) would not addCommand() for start_mysql since it is already
running. (Alternatively, mysql could be shutdown and then be restarted,
extending cluster creation time by some seconds.
The downside of this mechanism is that at least one service start occurs
in an unexpected place, and it does not work when multiple services must
be started, such as for multiple Zookeepers. Assuming the unexpected
place wart is acceptable, I could:
2. make Zookeeper init work by implementing a barrier of my own inside
ZooKeeperClusterActionHandler.afterConfiguration() of Zookeeper() that
uses a countdown latch to load data (properties) into ZK after all ZK
instances have started, but before any other service node is started.
"Run Whirr twice" approach:
3. First use Whirr to provision Zookeeper, and then a separate Whirr
run simply uses the ZK ensemble, loading property data once into ZK
afterConfiguration(). This approach unfortunately splits the whirr
state into 2 clusters, complicating network permissions, clusterIDs, and
shutdown/termination.
Extend Whirr to:
4. support more than one whirr.instance-templates (using a numeric
suffix like .0, .1, etc.) in a single run such that each instance
template is applied in suffix order in
ClusterController.launchCluster(ClusterSpec); this would mean only one
Cluster object would be created and there would be no unexpected place
wart. The implementation could either consider the templates to be
overlayed on each other (same number of commas required, noop is default
service) for the purposes of all phases except the start phase would
enforce an ordering.
Any thoughts/suggestions?
Paul
