[
https://issues.apache.org/jira/browse/JCLOUDS-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Hildrew updated JCLOUDS-490:
----------------------------------
Description:
Calls to retrieve the security groups in an Openstack tenant using the jClouds
ComputeService API is inefficient and very slow.
The following (scala) code is an example of a slow call:
val context = ContextBuilder.newBuilder("openstack-nova")
.endpoint(origin.endpoint)
.credentials(s"${origin.tenant}:${origin.user}", origin.secret)
.build(classOf[ComputeServiceContext])
val compute = context.getComputeService
val securityGroupExtension = compute.getSecurityGroupExtension.get
val secGroups = securityGroupExtension.listSecurityGroups
val result = secGroups.map ( fromJCloud(_, existingGroups) )
Most of this is very quick, but when debugging I the call to listSecurityGroups
is slow. In one example run this took 6 minutes and 17 seconds to retrieve 32
security groups. During this example call, 93 separate requests are made to the
Openstack API server. The first is a call to GET /v2/<id>/extensions and the
following 92 are calls to GET /v2/<id>/os-security-groups. These 92 calls all
get an identical response from the API and the calls are made with about 4
seconds delay. After all these calls the method returns with the correct
response.
One of the causes (but this doesn't seem to account for all the the excess
calls I'm seeing) is that the NovaSecurityGroupToSecurityGroup class makes
calls to transform each openstack SecurityGroupRule into a JClouds IpPermission
object. This is done in the SecurityGroupRuleToIpPermission class, which takes
a Predicate<AtomicReference<ZoneAndName>> and the supplied predicate is called
for every transformation that occurs in which the source is another security
group instead of a CIDR. The implementation of the predicate is
FindSecurityGroupWithNameAndReturnTrue - which makes an API request for the
list of security groups in a zone and then returns true if the supplied
reference is in the newly obtained list. The API call is made regardless of
whether it is the same zone or not.
In the particular zone I'm querying, there are a large number of rules (60)
that reference other security groups (all in the same zone). As I said, this
doesn't account for all the calls for exactly the same request - but it does
account for more than half of them. I think there must be another
transformation that also uses the predicate.
I note that the predicate actually modifies the object, which makes it trickier
to remove without changing the behaviour.
As a work around I am now using the NovaApi directly and can report that it
takes a few seconds, instead of the 6 minutes.
was:
Calls to retrieve the security groups in an Openstack tenant using the jClouds
ComputeService API is inefficient and very slow.
The following (scala) code is an example of a slow call:
{noformat}
val context = ContextBuilder.newBuilder("openstack-nova")
.endpoint(origin.endpoint)
.credentials(s"${origin.tenant}:${origin.user}", origin.secret)
.build(classOf[ComputeServiceContext])
val compute = context.getComputeService
val securityGroupExtension = compute.getSecurityGroupExtension.get
val secGroups = securityGroupExtension.listSecurityGroups
val result = secGroups.map ( fromJCloud(_, existingGroups) )
{noformat}
Most of this is very quick, but when debugging I the call to listSecurityGroups
is slow. In one example run this took 6 minutes and 17 seconds to retrieve 32
security groups. During this example call, 93 separate requests are made to the
Openstack API server. The first is a call to GET /v2/<id>/extensions and the
following 92 are calls to GET /v2/<id>/os-security-groups. These 92 calls all
get an identical response from the API and the calls are made with about 4
seconds delay. After all these calls the method returns with the correct
response.
One of the causes (but this doesn't seem to account for all the the excess
calls I'm seeing) is that the NovaSecurityGroupToSecurityGroup class makes
calls to transform each openstack SecurityGroupRule into a JClouds IpPermission
object. This is done in the SecurityGroupRuleToIpPermission class, which takes
a Predicate<AtomicReference<ZoneAndName>> and the supplied predicate is called
for every transformation that occurs in which the source is another security
group instead of a CIDR. The implementation of the predicate is
FindSecurityGroupWithNameAndReturnTrue - which makes an API request for the
list of security groups in a zone and then returns true if the supplied
reference is in the newly obtained list. The API call is made regardless of
whether it is the same zone or not.
In the particular zone I'm querying, there are a large number of rules (60)
that reference other security groups (all in the same zone). As I said, this
doesn't account for all the calls for exactly the same request - but it does
account for more than half of them. I think there must be another
transformation that also uses the predicate.
I note that the predicate actually modifies the object, which makes it trickier
to remove without changing the behaviour.
As a work around I am now using the NovaApi directly and can report that it
takes a few seconds, instead of the 6 minutes.
> Security group listing on nova slow
> -----------------------------------
>
> Key: JCLOUDS-490
> URL: https://issues.apache.org/jira/browse/JCLOUDS-490
> Project: jclouds
> Issue Type: Bug
> Affects Versions: 1.7.0, 1.7.1
> Environment: Linux, Scala, Openstack
> Reporter: Simon Hildrew
>
> Calls to retrieve the security groups in an Openstack tenant using the
> jClouds ComputeService API is inefficient and very slow.
> The following (scala) code is an example of a slow call:
> val context = ContextBuilder.newBuilder("openstack-nova")
> .endpoint(origin.endpoint)
> .credentials(s"${origin.tenant}:${origin.user}", origin.secret)
> .build(classOf[ComputeServiceContext])
> val compute = context.getComputeService
> val securityGroupExtension = compute.getSecurityGroupExtension.get
> val secGroups = securityGroupExtension.listSecurityGroups
> val result = secGroups.map ( fromJCloud(_, existingGroups) )
> Most of this is very quick, but when debugging I the call to
> listSecurityGroups is slow. In one example run this took 6 minutes and 17
> seconds to retrieve 32 security groups. During this example call, 93 separate
> requests are made to the Openstack API server. The first is a call to GET
> /v2/<id>/extensions and the following 92 are calls to GET
> /v2/<id>/os-security-groups. These 92 calls all get an identical response
> from the API and the calls are made with about 4 seconds delay. After all
> these calls the method returns with the correct response.
> One of the causes (but this doesn't seem to account for all the the excess
> calls I'm seeing) is that the NovaSecurityGroupToSecurityGroup class makes
> calls to transform each openstack SecurityGroupRule into a JClouds
> IpPermission object. This is done in the SecurityGroupRuleToIpPermission
> class, which takes a Predicate<AtomicReference<ZoneAndName>> and the supplied
> predicate is called for every transformation that occurs in which the source
> is another security group instead of a CIDR. The implementation of the
> predicate is FindSecurityGroupWithNameAndReturnTrue - which makes an API
> request for the list of security groups in a zone and then returns true if
> the supplied reference is in the newly obtained list. The API call is made
> regardless of whether it is the same zone or not.
> In the particular zone I'm querying, there are a large number of rules (60)
> that reference other security groups (all in the same zone). As I said, this
> doesn't account for all the calls for exactly the same request - but it does
> account for more than half of them. I think there must be another
> transformation that also uses the predicate.
> I note that the predicate actually modifies the object, which makes it
> trickier to remove without changing the behaviour.
> As a work around I am now using the NovaApi directly and can report that it
> takes a few seconds, instead of the 6 minutes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
