Hi Andrew, Sorry this took me so long to do - I didn't notice your last e-mail. It is now open at https://issues.apache.org/jira/browse/JCLOUDS-490 and thanks for your assistance :)
Simon -- Simon Hildrew Lead Infrastructure Developer theguardian.com +44 (0) 20 335 34173 On 20 February 2014 19:14, Andrew Bayer <[email protected]> wrote: > Yeah, I'd love it if you could open a JIRA - I'll try to make a run at it > the next week or two. Anything I can do to help my favorite newspaper. =) > > A. > > > On Thu, Feb 20, 2014 at 11:11 AM, Simon Hildrew < > [email protected]> wrote: > >> Hi Andrew, >> >> Thanks for replying - it has been a bit of a journey to identify the >> issue, and I'm afraid that whilst I can now find my way around the code >> base reasonably well - I'm not well versed enough to be able to attempt to >> create a fix without considerable further effort. On investigating further >> I realised that the predicate actually modifies the object, which makes it >> trickier to remove without changing the behaviour. >> >> I am now using the NovaApi directly in order to make it faster and can >> report that it takes a few seconds, instead of the 6 minutes. >> >> Unfortunately this means the AWS and OpenStack code is no longer shared >> (see >> https://github.com/guardian/prism/blob/master/app/collectors/securityGroup.scala). >> However, I'm already marshalling it into an object of my own, so I doubt it >> makes that much difference really. >> >> Would you like me to raise an issue for this? Would be keen to hear if it >> gets fixed at some point. >> >> Many thanks, >> Simon >> >> >> >> -- >> Simon Hildrew >> Lead Infrastructure Developer >> theguardian.com >> +44 (0) 20 335 34173 >> >> >> On 20 February 2014 18:10, Andrew Bayer <[email protected]> wrote: >> >>> Yipes, that's a bit inefficient. We could definitely use some >>> improvements there - I'll see if I can figure anything out. That said, if >>> you're only using one cloud (i.e., Nova), I'd say it's entirely reasonable >>> to just use the NovaApi security group calls directly - the point of the >>> SecurityGroupExtension at the compute abstraction level is to make >>> portability easier, but if you don't need to worry about portability, >>> you'll definitely get more control and efficiency out of the >>> provider-specific API calls... >>> >>> A. >>> >>> >>> On Mon, Feb 17, 2014 at 1:45 PM, Everett Toews < >>> [email protected]> wrote: >>> >>>> abayer, this is SecurityGroupExtension related. Do you have some time >>>> to look into it? >>>> >>>> Everett >>>> >>>> >>>> >>>> On Feb 17, 2014, at 1:10 PM, Simon Hildrew < >>>> [email protected]> wrote: >>>> >>>> On 3 February 2014 21:01, Simon Hildrew < >>>> [email protected]> wrote: >>>> >>>>> On 3 February 2014 20:51, Andrew Phillips <[email protected]>wrote: >>>>> >>>>>> Making a call to listSecurityGroupsInLocation and passing in the >>>>>>> single >>>>>>> zone I'm interested in has made no difference - it is still making >>>>>>> the >>>>>>> underlying call to the API many many times before the jclouds API >>>>>>> call >>>>>>> returns. It is always exactly 92 calls to the underlying API >>>>>>> >>>>>> >>>>>> Does this include things like authentication calls, or is this the >>>>>> "straight" call for security groups? If so, is there any obvious >>>>>> variation >>>>>> in parameters or other request characteristics? Or is it literally always >>>>>> the same call? >>>>>> >>>>> >>>>> There are two other calls to start with - one to keystone to auth >>>>> and then another to get the extensions in order to locate the security >>>>> group endpoint. The following 92 calls are always identical and get an >>>>> identical response each time (as observed via both tcpdump and today using >>>>> the jclouds.wire logging). It's really odd behaviour! >>>>> >>>>> >>>>>> Hopefully, one of the OpenStack experts on the list can shed more >>>>>> light on where these might be coming from. >>>>>> >>>>> >>>>> That would be great - I'm happy to post some of my logs or carry out >>>>> any experiments that might help shed more light. The problem seems to be >>>>> worse in some region / tenants than others so I'm going to try the same >>>>> experiment in some other places tomorrow and see if there is a pattern. >>>>> >>>> >>>> I've been continuing to try and shed more light on this, but have >>>> been super busy with other things over the last fortnight. >>>> >>>> I think I may have worked out what the root cause of the behaviour >>>> I'm seeing is, although I see no simple solution and would love some advice >>>> from the OpenStack experts. >>>> >>>> The initial list happens in one go. But in >>>> the NovaSecurityGroupToSecurityGroup class they make calls to transform >>>> each openstack SecurityGroupRule into a JClouds IpPermission object. This >>>> is done in the SecurityGroupRuleToIpPermission class, which takes >>>> a Predicate<AtomicReference<ZoneAndName>> and the supplied predicate is >>>> called for every transformation that occurs in which the source is another >>>> security group instead of a CIDR. The implementation of the predicate is >>>> FindSecurityGroupWithNameAndReturnTrue - which makes an API request for the >>>> list of security groups in a zone and then returns true if the supplied >>>> reference is in the newly obtained list. The API call is made regardless of >>>> whether it is the same zone or not. >>>> >>>> In the particular zone I'm querying, there are a large number of >>>> rules (60) that reference other security groups (all in the same zone). >>>> This doesn't account for the now 111 calls for exactly the same request - >>>> but it does account for more than half of them. I think there must be >>>> another transformation that also uses the predicate (I'd carry on looking >>>> but it's past time for me to go home). >>>> >>>> Part of me wonders whether I should use the NovaApi directly and do >>>> the translation myself. >>>> >>>> As always, any help or thoughts gratefully received. >>>> >>>> Thanks, >>>> Simon >>>> >>>> >>>> Please consider the environment before printing this email. >>>> ------------------------------------------------------------------ >>>> Visit theguardian.com >>>> >>>> On your mobile, download the Guardian iPhone app theguardian.com/iphone >>>> and our iPad edition theguardian.com/iPad >>>> Save up to 57% by subscribing to the Guardian and Observer - choose the >>>> papers you want and get full digital access. >>>> Visit subscribe.theguardian.com >>>> >>>> This e-mail and all attachments are confidential and may also >>>> be privileged. If you are not the named recipient, please notify >>>> the sender and delete the e-mail and all attachments immediately. >>>> Do not disclose the contents to another person. You may not use >>>> the information for any purpose, or store, or copy, it in any way. >>>> >>>> Guardian News & Media Limited is not liable for any computer >>>> viruses or other material transmitted with or as part of this >>>> e-mail. You should employ virus checking software. >>>> >>>> Guardian News & Media Limited >>>> >>>> A member of Guardian Media Group plc >>>> Registered Office >>>> PO Box 68164 >>>> Kings Place >>>> 90 York Way >>>> London >>>> N1P 2AP >>>> >>>> Registered in England Number 908396 >>>> >>>> -------------------------------------------------------------------------- >>>> >>>> >>>> >>>> >>> >> Please consider the environment before printing this email. >> ------------------------------------------------------------------ >> Visit theguardian.com >> >> On your mobile, download the Guardian iPhone app theguardian.com/iphone and >> our iPad edition theguardian.com/iPad >> Save up to 57% by subscribing to the Guardian and Observer - choose the >> papers you want and get full digital access. >> Visit subscribe.theguardian.com >> >> This e-mail and all attachments are confidential and may also >> be privileged. If you are not the named recipient, please notify >> the sender and delete the e-mail and all attachments immediately. >> Do not disclose the contents to another person. You may not use >> the information for any purpose, or store, or copy, it in any way. >> >> Guardian News & Media Limited is not liable for any computer >> viruses or other material transmitted with or as part of this >> e-mail. You should employ virus checking software. >> >> Guardian News & Media Limited >> >> A member of Guardian Media Group plc >> Registered Office >> PO Box 68164 >> Kings Place >> 90 York Way >> London >> N1P 2AP >> >> Registered in England Number 908396 >> >> -------------------------------------------------------------------------- >> >> >> > Please consider the environment before printing this email. ------------------------------------------------------------------ Visit theguardian.com On your mobile, download the Guardian iPhone app theguardian.com/iphone and our iPad edition theguardian.com/iPad Save up to 57% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.theguardian.com This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group plc Registered Office PO Box 68164 Kings Place 90 York Way London N1P 2AP Registered in England Number 908396 --------------------------------------------------------------------------
