Thanks for raising this Jose - sounds like some serious testing of
Brooklyn going on as well!
Richard's right: the AWS security groups seem to be eventually
consistent. Anecdotally, it can take up to 15 minutes for the API to
report that the security group is no longer in use. Within jclouds, we
retry the check+deletion a few times and then move on (so the security
group can be left behind). Given this is still happening, it means that
the eventual consistency is still there.
Improving things in jclouds is appealing, but tricky. The jclouds calls
would have to block for a long time or leave threads behind to clean up
after the delete request has returned. That would approach/philosophy of
the jclouds library, I believe. Perhaps we could tweak the retry times
though.
---
Andrew Kennedy wrote about this quite a while ago, and said about a tool
(based on jclouds) for deleting the security groups [1].
I know that some customer-specific work has run with that approach to
delete unused security groups periodically (that fit some naming
convention, and have been unused for at least some configurable length
of time).
---
We could potentially incorporate the async cleanup within Brooklyn.
I'm not sure whether that would just be background cleanup jobs within
Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
What do your code changes look like for deleting the security groups and
key-pairs?
Aled
[1]
http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-jclouds.html
On 30/11/2016 13:13, Richard Downer wrote:
Hi Jose,
This is challenging my memory somewhat, but I recall in the past that
jclouds had trouble with AWS security groups, because the AWS API would
report the security group as being "in use" and therefore undeletable, for
a considerable time after the VM using it was removed. So if there was an
attempt to delete the SG shortly after the VM was deleted, AWS would not
allow it.
I'm not sure if the situation has changed since then - either in the way
jclouds/Brooklyn handles it, or if the AWS API can now handle an SG delete
like this.
So I think there's some mileage in investigating this further, but also in
establishing the facts about the AWS API, and the history of this problem
(my memory is not reliable :-) before writing a lot of code!
And while I'm obviously happy to receive code contributions to Brooklyn,
it's also worth considering if any fixes that may be needed may be better
applied to the Apache jclouds project rather than here - then far more
people than the Brooklyn community would benefit.
Richard.
On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
Hi all,
During the last days I have provisioned about 1500 VM in AWS and Softlayer
using Brooklyn. (I am researching about large cloud-application
deployments).
However, I found a little problem working with AWS. When a VM is created,
security groups are also created, but when
the VM is deleted, the related security groups are not deleted. It was a
problem for me, because the limit of the security group pool is
500, and when the pool is full, new VM can not be provisiones because it
is not possible to add another security group.
Then, I think it could be good idea to allow security groups to be managed
by Brooklyn. jclouds provides some interesting interfaces
to manage security groups in a generic way, maintaing the current
abstraction in JcloudsLocation and JcloudsMachineLocation.
(Indeed, I already added to my code the capability of managing (deleting)
security groups (and keyPairs).)
does it make sense for you?
WDYT?
Thanks a lot,
Jose
