[
https://issues.apache.org/jira/browse/WHIRR-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tibor Kiss updated WHIRR-167:
-----------------------------
Attachment: whirr-167-2.patch
I attached my second patch which for now has a two level of parallelization.
There is a Callable per roles, each runs a StartupProcess.
Inside the StartupProcess is handled the retry mechanism and failed nodes
cleanup. The StartupProcess also uses a Callable for starting instances and
also for cleaning up the failed nodes.
In order to be able to JUnit tests using mockito, I had to extract some static
constructs, for example the creation of ClusterActionHandler map is factored
outside of ScriptBasedClusterAction. Also ComputeServiceContextBuilder got a
new method to be able to feed a ComputeServiceContextFactory from the outside
of ScriptBasedClusterAction. In this mode it is possible to JUnit test the
internal algorithm of BootstrapClusterAction.
Regarding the OptionSpec and hierarchical properties, I had to do it something
similar with the instance-templates, in the sense that roles may appear only in
the value side, as I mentioned in my previouse comment.
Please review it again.
> Improve bootstrapping and configuration to be able to isolate and repair or
> evict failing nodes on EC2
> ------------------------------------------------------------------------------------------------------
>
> Key: WHIRR-167
> URL: https://issues.apache.org/jira/browse/WHIRR-167
> Project: Whirr
> Issue Type: Improvement
> Environment: Amazon EC2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Attachments: whirr-167-1.patch, whirr-167-2.patch, whirr.log
>
>
> Actually it is very unstable the cluster startup process on Amazon EC2
> instances. How the number of nodes to be started up is increasing the startup
> process it fails more often. But sometimes even 2-3 nodes startup process
> fails. We don't know how many number of instance startup is going on at the
> same time at Amazon side when it fails or when it successfully starting up.
> The only think I see is that when I am starting around 10 nodes, the
> statistics of failing nodes are higher then with smaller number of nodes and
> is not direct proportional with the number of nodes, looks like it is
> exponentialy higher probability to fail some nodes.
> Lookint into BootstrapCluterAction.java, there is a note "// TODO: Check for
> RunNodesException and don't bail out if only a few " which indicated the
> current unreliable startup process. So we should improve it.
> We could add a "max percent failure" property (per instance template), so
> that if the number failures exceeded this value the whole cluster fails to
> launch and is shutdown. For the master node the value would be 100%, but for
> datanodes it would be more like 75%. (Tom White also mentioned in an email).
> Let's discuss if there are any other requirements to this improvement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
