Re: AWS Request Rate
On Thu, Feb 20, 2014 at 12:02 PM, David Britton wrote: > "2": > agent-state-info: '(error: cannot set up groups: Request limit > exceeded. (RequestLimitExceeded))' As a side note, I'd just like to congratulate the juju team for offering such a clear error message in the right place. gustavo @ http://niemeyer.net -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
We also ended up with an instance updater that makes a request on a pretty fast poll when you bring new instances up. It should be changed to poll for all unknown instances in a single bulk request. Possibly the poll instance should also be decreased. (I triggered it with just deploy -n 15, and as early as the third instance would fail) Also, Limit Exceeded should be treated as a temporary failure. I'm ok if we have other failures to provision be fatal, but that one is pretty obvious. John =:-> On Feb 21, 2014 1:10 AM, "David Cheney" wrote: > It could be related, I know that John, Andrew and Dimiter landed some > fixes a while back that made juju more frugal with it's AWS query usage. > > > On Fri, Feb 21, 2014 at 7:32 AM, David Britton < > david.brit...@canonical.com> wrote: > >> On Fri, Feb 21, 2014 at 06:59:53AM +1100, David Cheney wrote: >> > Which version of Juju. I could get juju into state pretty easily a few >> > months ago >> > >> >> Interestingly enough, I just upgraded to trusty (1.17 juju line) and >> noticed the >> significant speed improvements in contacting AWS. I'm wondering if that >> is responsible for me hitting this request rate limit. >> >> -- >> David Britton >> > > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
It could be related, I know that John, Andrew and Dimiter landed some fixes a while back that made juju more frugal with it's AWS query usage. On Fri, Feb 21, 2014 at 7:32 AM, David Britton wrote: > On Fri, Feb 21, 2014 at 06:59:53AM +1100, David Cheney wrote: > > Which version of Juju. I could get juju into state pretty easily a few > > months ago > > > > Interestingly enough, I just upgraded to trusty (1.17 juju line) and > noticed the > significant speed improvements in contacting AWS. I'm wondering if that > is responsible for me hitting this request rate limit. > > -- > David Britton > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
On Fri, Feb 21, 2014 at 06:59:53AM +1100, David Cheney wrote: > Which version of Juju. I could get juju into state pretty easily a few > months ago > Interestingly enough, I just upgraded to trusty (1.17 juju line) and noticed the significant speed improvements in contacting AWS. I'm wondering if that is responsible for me hitting this request rate limit. -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
Which version of Juju. I could get juju into state pretty easily a few months ago On Fri, Feb 21, 2014 at 2:45 AM, David Britton wrote: > On Thu, Feb 20, 2014 at 10:37:44AM -0500, Kapil Thangavelu wrote: > > i filed a bug against core for this a few months back > > https://bugs.launchpad.net/juju-core/+bug/1227450 > > Thanks -- > > I'll append this specific AWS case to that bug... Just to capture. > > -- > David Britton > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
On Thu, Feb 20, 2014 at 10:37:44AM -0500, Kapil Thangavelu wrote: > i filed a bug against core for this a few months back > https://bugs.launchpad.net/juju-core/+bug/1227450 Thanks -- I'll append this specific AWS case to that bug... Just to capture. -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
i filed a bug against core for this a few months back https://bugs.launchpad.net/juju-core/+bug/1227450 pyjuju did indeed retry. -k On Thu, Feb 20, 2014 at 10:35 AM, David Britton wrote: > On Thu, Feb 20, 2014 at 10:12:13AM -0500, Kapil Thangavelu wrote: > > > > 2. afaicr for a given failed provisioning request, there is no retry in > > core. > > Thanks Kapil and Marco for the replies > > Would this be worth putting in a bug report about? Or is it working as > desired? > > I seem to remember pyjuju being more resilient in this area. > > > -- > David Britton > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
Hello David, When I first read your email, it reminded me of a bug ( https://bugs.launchpad.net/juju-core/+bug/1279879 ) that I opened where I got the error message: "Maximum number of attempts (3) reached sending request..." but upon further review your problem appears to be a different issue. - Matt On Thu, Feb 20, 2014 at 9:02 AM, David Britton wrote: > Hi Juju Folks -- > > I've been getting this from time to time: > > "2": > agent-state-info: '(error: cannot set up groups: Request limit > exceeded. (RequestLimitExceeded))' > instance-id: pending > series: precise > > Which lead me to this: > > > http://docs.aws.amazon.com/AWSEC2/latest/APIReference/api-error-codes.html > > Which says this: > > Amazon EC2 API requests are throttled to help maintain the performance > of the service. If your requests have been throttled, you'll get the > following error: Client.RequestLimitExceeded. For more information, see > Query API Request Rate in the Amazon Elastic Compute Cloud User Guide. > > > Some questions: > > 1) Is there something I can do to prevent this? I'm deploying a stack > with juju-deployer. > > 2) When it happens, it seems that juju is dead in the water on that > operation (in this case launching an instnace). Should it instead > throttle back and keep trying? I have to terminate-machine, > destroy-service, and re-run deployer (which thankfully picks up where it > left off). Is that expected behavior for the juju system? > > 3) This is a shared Amazon account, is this requestLimit across all of > the people that have api credentials, or just me? > > > Thanks! > > -- > David Britton > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > -- - Matt -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
On Thu, Feb 20, 2014 at 10:12:13AM -0500, Kapil Thangavelu wrote: > > 2. afaicr for a given failed provisioning request, there is no retry in > core. Thanks Kapil and Marco for the replies Would this be worth putting in a bug report about? Or is it working as desired? I seem to remember pyjuju being more resilient in this area. -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: AWS Request Rate
1) try firewall-mode: global juju is trying to create sec groups per machine, which hopefully can get better with a move to vpc (dynamic sec groups on instances). That should reduce aws api usage a bit for machine provisioning. 2. afaicr for a given failed provisioning request, there is no retry in core. 3. its across all users on the same account against a given region. if your using iam for separate users, the rate limit is still against the underlying account., each api has its own throttle rate. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/query-api-troubleshooting.html -k On Thu, Feb 20, 2014 at 10:02 AM, David Britton wrote: > Hi Juju Folks -- > > I've been getting this from time to time: > > "2": > agent-state-info: '(error: cannot set up groups: Request limit > exceeded. (RequestLimitExceeded))' > instance-id: pending > series: precise > > Which lead me to this: > > > http://docs.aws.amazon.com/AWSEC2/latest/APIReference/api-error-codes.html > > Which says this: > > Amazon EC2 API requests are throttled to help maintain the performance > of the service. If your requests have been throttled, you'll get the > following error: Client.RequestLimitExceeded. For more information, see > Query API Request Rate in the Amazon Elastic Compute Cloud User Guide. > > > Some questions: > > 1) Is there something I can do to prevent this? I'm deploying a stack > with juju-deployer. > > 2) When it happens, it seems that juju is dead in the water on that > operation (in this case launching an instnace). Should it instead > throttle back and keep trying? I have to terminate-machine, > destroy-service, and re-run deployer (which thankfully picks up where it > left off). Is that expected behavior for the juju system? > > 3) This is a shared Amazon account, is this requestLimit across all of > the people that have api credentials, or just me? > > > Thanks! > > -- > David Britton > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
AWS Request Rate
Hi Juju Folks -- I've been getting this from time to time: "2": agent-state-info: '(error: cannot set up groups: Request limit exceeded. (RequestLimitExceeded))' instance-id: pending series: precise Which lead me to this: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/api-error-codes.html Which says this: Amazon EC2 API requests are throttled to help maintain the performance of the service. If your requests have been throttled, you'll get the following error: Client.RequestLimitExceeded. For more information, see Query API Request Rate in the Amazon Elastic Compute Cloud User Guide. Some questions: 1) Is there something I can do to prevent this? I'm deploying a stack with juju-deployer. 2) When it happens, it seems that juju is dead in the water on that operation (in this case launching an instnace). Should it instead throttle back and keep trying? I have to terminate-machine, destroy-service, and re-run deployer (which thankfully picks up where it left off). Is that expected behavior for the juju system? 3) This is a shared Amazon account, is this requestLimit across all of the people that have api credentials, or just me? Thanks! -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
