This litter is irritating. On the other hand we don't want application stops to wait an arbitrarily long time for sec groups to become deletable.
I'd love to switch to a model where location instances become entities. Security groups could also become part of that, and managed explicitly in the runtime model. "Stop" might then take a long time, or fail, but at least it's explicit... Or if we have some kind of internal "application" with management state, it might have a "recycling bin" and as part of "stop" we move redundant but not-yet-deletable security groups there, and a policy there tries periodically to remove them. This way it's transparent what we're doing, but doesn't get in the way of normal operation, and also doesn't leave litter behind. (BTW @jose for your use case maybe it's an option to tell brooklyn to use a pre-existing security group?) Best Alex On 1 December 2016 at 10:20, Geoff Macartney < geoff.macart...@cloudsoftcorp.com> wrote: > Note the PR "Factor security group code out into separate utility class" > [1] from last week, might be of some use here, but it just provides a basic > toolkit for security groups (not keys yet, but they could be added?). > > Factor out the functionality for security groups. > > Brings the create/delete and add/remove permissions functionality > for security groups from JcloudsLocationSecurityGroupCustomizer > out into its own class. The aim is to make it more convenient > to write other customizers etc. that might need security groups. > > This is intended as nearly as possible as a behaviour-preserving > change. I have permitted myself one change to the behaviour where > this seemed sensible; the loop in runOperationWithRetry > was effectively an infinite loop because of its 100 iteration > exponential back-off, so I have changed this to a back-off that > tries for about five minutes. > > For the security group deletion, what do you think of the suggestion of > just changing `removeSecurityGroup` [2] such that `runOperationWithRetry` > will explicitly do an exponential back-off? (i.e. at present it retries > depending on the `isExceptionRetryable` value but it could override that.) > > Or is it better to have an explicit long-lived service within Brooklyn > dedicated to that purpose? > > Geoff > > > [1] https://github.com/apache/brooklyn-server/pull/425 > [2] > https://github.com/apache/brooklyn-server/blob/master/ > locations/jclouds/src/main/java/org/apache/brooklyn/ > location/jclouds/networking/SecurityGroupEditor.java#L114-L123 > > On Wed, 30 Nov 2016 at 14:20 Aled Sage <aled.s...@gmail.com> wrote: > > > Thanks for raising this Jose - sounds like some serious testing of > > Brooklyn going on as well! > > > > Richard's right: the AWS security groups seem to be eventually > > consistent. Anecdotally, it can take up to 15 minutes for the API to > > report that the security group is no longer in use. Within jclouds, we > > retry the check+deletion a few times and then move on (so the security > > group can be left behind). Given this is still happening, it means that > > the eventual consistency is still there. > > > > Improving things in jclouds is appealing, but tricky. The jclouds calls > > would have to block for a long time or leave threads behind to clean up > > after the delete request has returned. That would approach/philosophy of > > the jclouds library, I believe. Perhaps we could tweak the retry times > > though. > > > > --- > > Andrew Kennedy wrote about this quite a while ago, and said about a tool > > (based on jclouds) for deleting the security groups [1]. > > > > I know that some customer-specific work has run with that approach to > > delete unused security groups periodically (that fit some naming > > convention, and have been unused for at least some configurable length > > of time). > > > > --- > > We could potentially incorporate the async cleanup within Brooklyn. > > > > I'm not sure whether that would just be background cleanup jobs within > > Brooklyn, or whether we'd somehow expose this on the Brooklyn API. > > > > What do your code changes look like for deleting the security groups and > > key-pairs? > > > > Aled > > > > [1] > > > > http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after- > jclouds.html > > > > > > On 30/11/2016 13:13, Richard Downer wrote: > > > Hi Jose, > > > > > > This is challenging my memory somewhat, but I recall in the past that > > > jclouds had trouble with AWS security groups, because the AWS API would > > > report the security group as being "in use" and therefore undeletable, > > for > > > a considerable time after the VM using it was removed. So if there was > an > > > attempt to delete the SG shortly after the VM was deleted, AWS would > not > > > allow it. > > > > > > I'm not sure if the situation has changed since then - either in the > way > > > jclouds/Brooklyn handles it, or if the AWS API can now handle an SG > > delete > > > like this. > > > > > > So I think there's some mileage in investigating this further, but also > > in > > > establishing the facts about the AWS API, and the history of this > problem > > > (my memory is not reliable :-) before writing a lot of code! > > > > > > And while I'm obviously happy to receive code contributions to > Brooklyn, > > > it's also worth considering if any fixes that may be needed may be > better > > > applied to the Apache jclouds project rather than here - then far more > > > people than the Brooklyn community would benefit. > > > > > > Richard. > > > > > > > > > > > > On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote: > > > > > >> Hi all, > > >> > > >> During the last days I have provisioned about 1500 VM in AWS and > > Softlayer > > >> using Brooklyn. (I am researching about large cloud-application > > >> deployments). > > >> However, I found a little problem working with AWS. When a VM is > > created, > > >> security groups are also created, but when > > >> the VM is deleted, the related security groups are not deleted. It > was a > > >> problem for me, because the limit of the security group pool is > > >> 500, and when the pool is full, new VM can not be provisiones because > it > > >> is not possible to add another security group. > > >> > > >> Then, I think it could be good idea to allow security groups to be > > managed > > >> by Brooklyn. jclouds provides some interesting interfaces > > >> to manage security groups in a generic way, maintaing the current > > >> abstraction in JcloudsLocation and JcloudsMachineLocation. > > >> (Indeed, I already added to my code the capability of managing > > (deleting) > > >> security groups (and keyPairs).) > > >> > > >> does it make sense for you? > > >> WDYT? > > >> > > >> Thanks a lot, > > >> Jose > > >> > > >> > > >> > > >> > > > > > > >
