On 20121211 1:39 , Andrei Savu wrote:
Hi Paul -
I would go for the "Run Whirr multiple times" approach. If you adress the
following issue you should be able to run all the instances in the same
security group with unrestricted connectivity.
https://issues.apache.org/jira/browse/WHIRR-336
I added a question about the exact requirements.
And to simplify things you can write a customised launcher that will take
care of allocating cluster names, executing multiple launchCluster commands
and shutting down everything as expected.
My plan is to add a new service that can be added to the trunk, so it
needs to work via the command line. Is there a way to use a customized
launcher via standard Whirr command line?
Here is a sketch of a variation on the 4th approach ("Whirr Waves"):
Starting with an example of a two "wave" (avoiding "phase" which already
has a meaning) cluster:
whirr.instance-templates.0=1 zookeeper, 2 zookeeper, 4 noop
The .0 templates run like normal during "whirr launch-cluster" and then,
during the same Whirr run, the .1 template is applied as a modification
of the same cluster so that new host instances are not allocated:
whirr.instance-templates.1=1 my-master, 2 my-gateway, 4 my-worker
In the second wave, instance provisioning is inhibited, the
instance-templates must have the same number of commas and same sequence
of leading numbers, and a role is only allowed to appear in one wave.
This might seem contrived, but here is another example:
whirr.instance-templates.0=1 mysql-master+zookeeper, 1
mysql-master+zookeeper,4 noop
whirr.instance-templates.1=1 hadoop-namenode+hadoop-jobtracker, 1
hbase-master+hadoop-secondarynn,4
hadoop-tasktracker+hadoop-datanode+hbase-regionserver
In the first wave, the two mysql-masters form a multi-master ensemble
which keeps state information about the setup in the Whirr process (much
like ZooKeeperCluster.getHosts(cluster) informs services/hbase of the
quorum).
In the second wave, nodes are not allocated
(BootstrapClusterAction.doAction() is inhibited if instances already
exist), but all phases for LaunchClusterCommand, including
beforeBootstrap() and afterBootstrap() callbacks, are executed. If
other whirr.cli.command.*Command are run, they would see a combined
whirr.instance-templates that works like normal.
Obviously, there would be no reason to limit this to 2 waves, but I do
not expect more than 10 waves to be useful, so the pattern could enable
a suffix of "\.[0-9]".
If sufficient state is stored and the number of instances of a service
template were flexible, this is a stepping stone for a way to support a
Whirr modify-cluster command that can add services and change number of
instances implementing a service template.
Paul
-- Andrei Savu / axemblr.com
On Tue, Dec 11, 2012 at 5:25 AM, Paul Baclace <[email protected]>wrote:
I have a use case where I want to start up some services in a particular
order. I am using Zookeeper to store cluster-wide properties that need to
be set before dependent services start. (A shared dbms that needs some
specific records set before worker nodes start presents a similar
situation, for example: extending services/hbase to use MySQL for
metadata.) What is the right way to do this in Whirr? My current ideas
are numbered below.
Looking at ClusterActionHandlerSupport (action callbacks) and:
https://cwiki.apache.org/**confluence/display/WHIRR/**
Implementing+a+New+Service<https://cwiki.apache.org/confluence/display/WHIRR/Implementing+a+New+Service>
which states that phases of a Whirr run are essentially cluster-wide
synchronization barriers (bootstrap, configuration, and I assume also
start, stop, cleanup, destroy), but there is no way to specify order within
a phase; I see that:
1. There is a way to perform an action afterConfiguration() which occurs
prior to any calls to beforeStart(), as long as a kind of daemon only
occurs once per cluster (like a master node for HBase) so these dependent
actions would only happen once. This works okay for the situation where
one MySQL could be started up, a new user and db is created, and then the
beforeStart() for mysql (if mysql was a Whirr service) would not
addCommand() for start_mysql since it is already running. (Alternatively,
mysql could be shutdown and then be restarted, extending cluster creation
time by some seconds.
The downside of this mechanism is that at least one service start occurs
in an unexpected place, and it does not work when multiple services must be
started, such as for multiple Zookeepers. Assuming the unexpected place
wart is acceptable, I could:
2. make Zookeeper init work by implementing a barrier of my own inside
ZooKeeperClusterActionHandler.**afterConfiguration() of Zookeeper() that
uses a countdown latch to load data (properties) into ZK after all ZK
instances have started, but before any other service node is started.
"Run Whirr twice" approach:
3. First use Whirr to provision Zookeeper, and then a separate Whirr run
simply uses the ZK ensemble, loading property data once into ZK
afterConfiguration(). This approach unfortunately splits the whirr state
into 2 clusters, complicating network permissions, clusterIDs, and
shutdown/termination.
Extend Whirr to:
4. support more than one whirr.instance-templates (using a numeric suffix
like .0, .1, etc.) in a single run such that each instance template is
applied in suffix order in ClusterController.**launchCluster(ClusterSpec);
this would mean only one Cluster object would be created and there would be
no unexpected place wart. The implementation could either consider the
templates to be overlayed on each other (same number of commas required,
noop is default service) for the purposes of all phases except the start
phase would enforce an ordering.
Any thoughts/suggestions?
Paul
