[
https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173492#comment-13173492
]
Andrei Savu commented on WHIRR-455:
-----------------------------------
Thanks Bob for opening this JIRA. We are trying to provide an workaround for
this in 0.8.0 by implementing the ability to resize existing clusters. This
should allow you to go from 5 to 50 by adding 5 new machines on each iteration.
Sounds reasonable? Do you think this approach could work for you?
Also see this page http://www.jclouds.org/documentation/reference/pool-design
for some of the things we are working on in jclouds.
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9
> 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot
> instances.
> It will eventually fail trying to poll the security groups for the pending
> instances, and eventually ec2 will block the requests due to the number of
> requests.
> The spot bids will be created, and instances will be started in them, but
> whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a
> jclouds issue, but I figured I'd start here.
> configuration file:
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50
> hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment
> variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See
> http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
