Hi Nirmal, I did some further debugging on this issue and confirmed that the new jClouds property mentioned below actually took effect. However, with or without this new jClouds property enabled, the floating allocation process fails when we invoke FloatinIPApi.create() method. This is due to a known jClouds issue that has not been fixed according to https://issues.apache.org/jira/browse/JCLOUDS-471.
The problem is that at the Openstack level, when you create a floating IP, you need to supply the name of the floating IP pool. I think create() used to work with older version of Openstack where if floating IP pool info is not supplied, Openstack will query the default configured value. Apparently, this is no longer the case with Havana. To fix this issue, I used FloatingIPApi.allocateFromPool(defaultFloatingIpPool) instead of create(). This allows us to be more explicit as to which floating IP pool to use. I'm introducing a new property named "defaultFloatingIpPool" in cloud-controller.xml under IaasProvider section. This new property is used as the default floating IP pool for the Iaas provider. If a cartridge needs to override this, we can change the value in the cartridge definition, under the IaasProvider section. I think not only the above will fix the current issue we're having with floating IP allocation, it will also give us some flexibility in terms of being able to explicitly specify which floating IP pool to be used. I made it so that If the new property is defined, I use FloatingIPApi.allocateFromPool(). Otherwise, I use the existing API FloatingIPApi.create(). I've tested and confirmed that with this fix, the new instance is spawned and becomes active without the pre-allocated floating IP. WDYT? -Jeffrey From: Nirmal Fernando <nir...@wso2.com<mailto:nir...@wso2.com>> Date: Monday, May 12, 2014 9:31 PM To: jeffrngu <jeffr...@cisco.com<mailto:jeffr...@cisco.com>> Cc: "dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>" <dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>> Subject: Re: Issue with floating IP allocation On Tue, May 13, 2014 at 9:44 AM, Jeffrey Nguyen (jeffrngu) <jeffr...@cisco.com<mailto:jeffr...@cisco.com>> wrote: Thank you Nirmal. Please see my responses under [Jeffrey] Also, in my previous email I had a question about why this new jClouds property along with the existing ones are not showing up in the payload for cartridge deployment. Jclouds wire log would not show these, these are used by CC and jclouds for internal operations. These doesn't make sense to pass to the instance. I'm trying to confirm if the properties defined under Iaas provider are sent to jClouds. Can you take a look? Thanks, -Jeffrey From: Nirmal Fernando <nirmal070...@gmail.com<mailto:nirmal070...@gmail.com>> Reply-To: "dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>" <dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>> Date: Saturday, May 10, 2014 7:11 PM To: "dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>" <dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>> Cc: Nirmal Fernando <nir...@wso2.com<mailto:nir...@wso2.com>> Subject: Re: Issue with floating IP allocation Hi Jeffrey, On Fri, May 9, 2014 at 12:59 AM, Jeffrey Nguyen (jeffrngu) <jeffr...@cisco.com<mailto:jeffr...@cisco.com>> wrote: Copied the right email address… From: jeffrngu <jeffr...@cisco.com<mailto:jeffr...@cisco.com>> Date: Thursday, May 8, 2014 12:27 PM To: "dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>" <dev@stratos.incubator.apache.org<mailto:dev@stratos.incubator.apache.org>>, "Nirmal Fernando (JIRA)" <j...@apache.org<mailto:j...@apache.org>> Subject: Issue with floating IP allocation Hi Nirmal, Not sure when we resolve email issue with Stratos alias so I copied you explicitly on this email. Anyway, I opened STRATOS-646 yesterday to track this issue where floating IP allocation fails when we don't pre-allocate the floating IP. I was doing some experiment with a jClouds property named "jclouds.openstack-nova.auto-create-floating-ips". I put it inside the property section where we define the "autoAssignIp" parameter as shown in [1] below. When I tried to subscribe to this cartridge, I got some exceptions shown in [2] below. The place you have used this property is correct. But you have to set autoAssignIp too, to true. [Jeffrey] I tested again with autoAsignIp set to true in addition to the new jClouds property but I still saw the same problem. Any other suggestions? From the java-doc description for the jclouds property, it looks like it is used to create floating IP on-demand AND assign the same floating IP to the spawned instance. For our case, sometimes we want to assign our own floating IP, but we just need it to be allocated at the Iaas level without us manually creating it. I'm not sure this jClouds property was designed to handle this case. I remember Subbu had the same confusions and we managed to test it successfully. Please correct me if I'm wrong on this. Based on the implementation for "ComputeServiceBuilderUtil.buildDefaultComputeService()", we're taking the properties defined in the "property" section of cartridge definition and send it to jClouds as-is. However, in the jClouds wire log, I don't see them in the server create request being sent to jClouds. Or are they encrypted inside "user_data"? Basically, I was trying to see if the above jClouds setting can solve this issue with floating IP allocation. If it does, the next question would be whether we use it as-is or we define a more user-friendly property for Stratos and then internally mask it out with the actual jClouds property. Proper fix would be to set this property if and only if autoAssignIp set to true, in the OpenstackNovaIaas. Then we do not need to specify this jclouds property explicitly. Could you submit a patch? :-) [Jeffrey] I can work on this. However, I think we should make sure the new jClouds property can fix this issue before we worry about how to integrate it into Stratos code. Please let me know what you think. Thanks, -Jeffrey [1] Modified cartridge def "iaasProvider": [ { "imageId": "RegionOne/a00d6082-61c8-4a4c-be3d-9ffbf3968792", "maxInstanceLimit": "4", "networkInterfaces": [ { "networkUuid": "de9464d6-a5c7-4e37-8842-602065a2546a", } ], "property": [ { "name": "instanceType", "value": "RegionOne/a033a14f-ad30-4703-a5bd-ac6496e1d580" }, { "name": "keyPair", "value": "phoenix" }, { "name": "autoAssignIp", "value": "false" }, { "name": "jclouds.openstack-nova.auto-create-floating-ips", "value": "true" }, { "name": "securityGroups", "value": "default" } ], "type": "openstack-Core" } [2] Snippet from wso2carbon.log TID: [0] [STRATOS] [2014-05-08 10:44:02,184] DEBUG {jclouds.wire} - >> "{"server":{"name":"cisco-samp-980","imageRef":"a00d6082-61c8-4a4c-be3d-9ffbf3968792","flavorRef":"a033a14f-ad30-4703-a5bd-ac6496e1d580","metadata":{"jclouds-group":"cisco-samp"},"key_name":"phoenix","security_groups":[{"name":"default"}],"user_data":"U0VSVklDRV9OQU1FPWNpc2NvLXNhbXBsZS12bSxIT1NUX05BTUU9Y2lzY28tc2FtcGxlLXZtLnFtb2cuY2lzY28uY29tLE1VTFRJVEVOQU5UPWZhbHNlLFRFTkFOVF9JRD0tMTIzNCxURU5BTlRfUkFOR0U9LTEyMzQsQ0FSVFJJREdFX0FMSUFTPWNpc2NvLXNhbXBsZS12bSxDTFVTVEVSX0lEPWNpc2NvLXNhbXBsZS12bS5jaXNjby1zYW1wbGUtdixDQVJUUklER0VfS0VZPXBTQ1hCdmk2QXBxME5TY2MsREVQTE9ZTUVOVD1kZWZhdWx0LFJFUE9fVVJMPWh0dHA6Ly9vY3RsLTAxLnFtb2cuY2lzY28uY29tOjEwMDgwL2dpdC9kZWZhdWx0LmdpdCxQT1JUUz0yMixQVVBQRVRfSVA9UFVQUEVUX0lQLFBVUFBFVF9IT1NUTkFNRT1QVVBQRVRfSE9TVE5BTUUsUFVQUEVUX0VOVj1QVVBQRVRfRU5WLE1PTklUT1JJTkdfU0VSVkVSX0FETUlOX1VTRVJOQU1FPXh4eHgsTU9OSVRPUklOR19TRVJWRVJfU0VDVVJFX1BPUlQ9MCxDRVJUX1RSVVNUU1RPUkU9L29wdC9hcGFjaGUtc3RyYXRvcy1jYXJ0cmlkZ2UtYWdlbnQvc2VjdXJpdHkvY2xpZW50LXRydXN0c3RvcmUuamtzLE1PTklUT1JJTkdfU0VSVkVSX0lQPW9jdGwtMDEucW1vZy5jaXNjby5jb20sTUJfSVA9b2N0bC0wMS5xbW9nLmNpc2NvLmNvbSxNQl9QT1JUPTYxNjE2LENFUF9QT1JUPTc2MTEsTU9OSVRPUklOR19TRVJWRVJfQURNSU5fUEFTU1dPUkQ9eHh4eCxDRVBfSVA9b2N0bC0wMS5xbW9nLmNpc2NvLmNvbSxTSU1QTEVfUFJPUEVSVFk9dmFsdWUsTU9OSVRPUklOR19TRVJWRVJfUE9SVD0wLEVOQUJMRV9EQVRBX1BVQkxJU0hFUj1mYWxzZSxUUlVTVFNUT1JFX1BBU1NXT1JEPXdzbzJjYXJib24sQ09NTUlUX0VOQUJMRUQ9ZmFsc2UsTUVNQkVSX0lEPWNpc2NvLXNhbXBsZS12bS5jaXNjby1zYW1wbGUtdjgwZTY1ZDI5LTAzMmQtNGNkNy05NWQ1LTg1YmVjZGYyZTZjOCxMQl9DTFVTVEVSX0lEPW51bGwsTkVUV09SS19QQVJUSVRJT05fSUQ9TjEsUEFSVElUSU9OX0lEPVJlZ2lvbk9uZQ\u003d\u003d","networks":[{"uuid":"de9464d6-a5c7-4e37-8842-602065a2546a"}]}}" {jclouds.wire} TID: [0] [STRATOS] [2014-05-08 10:44:02,707] DEBUG {jclouds.wire} - << "{"server": {"security_groups": [{"name": "default"}], "OS-DCF:diskConfig": "MANUAL", "id": "f0a4a1ae-1a7f-418f-a629-423a1f5ff998", "links": [{"href": "http://10.94.251.12:8774/v2/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "self"}, {"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "bookmark"}], "adminPass": "P6BA3jPCPPzB"}}" {jclouds.wire} TID: [0] [STRATOS] [2014-05-08 10:44:02,854] DEBUG {jclouds.wire} - << "{"server": {"status": "BUILD", "updated": "2014-05-08T10:44:02Z", "hostId": "", "OS-EXT-SRV-ATTR:host": null, "addresses": {}, "links": [{"href": "http://10.94.251.12:8774/v2/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "self"}, {"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "bookmark"}], "key_name": "phoenix", "image": {"id": "a00d6082-61c8-4a4c-be3d-9ffbf3968792", "links": [{"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/images/a00d6082-61c8-4a4c-be3d-9ffbf3968792";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": "scheduling", "OS-EXT-STS:vm_state": "building", "OS-EXT-SRV-ATTR:instance_name": "instance-000001af", "OS-SRV-USG:launched_at": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": null, "flavor": {"id": "a033a14f-ad30-4703-a5bd-ac6496e1d580", "links": [{"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/flavors/a033a14f-ad30-4703-a5bd-ac6496e1d580";, "rel": "bookmark"}]}, "id": "f0a4a1ae-1a7f-418f-a629-423a1f5ff998", "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "bfb57e32b9c941608112db017ba9248f", "name": "cisco-samp-980", "created": "2014-05-08T10:44:02Z", "tenant_id": "161d4e9387704433a8519259d65186ba", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 0, "config_drive": "", "metadata": {"jclouds-group": "cisco-samp"}}}" {jclouds.wire} …. TID: [0] [STRATOS] [2014-05-08 10:44:07,953] DEBUG {jclouds.wire} - << "{"server": {"status": "ACTIVE", "updated": "2014-05-08T10:44:07Z", "hostId": "30b72df643a1170bf586d61c107d73c5efee42f812e95f6c4a98663d", "OS-EXT-SRV-ATTR:host": "denver-stack03", "addresses": {"core": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:ac:28:59", "version": 4, "addr": "172.16.2.17", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.94.251.12:8774/v2/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "self"}, {"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/servers/f0a4a1ae-1a7f-418f-a629-423a1f5ff998";, "rel": "bookmark"}], "key_name": "phoenix", "image": {"id": "a00d6082-61c8-4a4c-be3d-9ffbf3968792", "links": [{"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/images/a00d6082-61c8-4a4c-be3d-9ffbf3968792";, "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-000001af", "OS-SRV-USG:launched_at": "2014-05-08T10:44:06.000000", "OS-EXT-SRV-ATTR:hypervisor_hostname": "denver-stack03.cisco.com<http://denver-stack03.cisco.com>", "flavor": {"id": "a033a14f-ad30-4703-a5bd-ac6496e1d580", "links": [{"href": "http://10.94.251.12:8774/161d4e9387704433a8519259d65186ba/flavors/a033a14f-ad30-4703-a5bd-ac6496e1d580";, "rel": "bookmark"}]}, "id": "f0a4a1ae-1a7f-418f-a629-423a1f5ff998", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "AZ-1", "user_id": "bfb57e32b9c941608112db017ba9248f", "name": "cisco-samp-980", "created": "2014-05-08T10:44:02Z", "tenant_id": "161d4e9387704433a8519259d65186ba", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {"jclouds-group": "cisco-samp"}}}" {jclouds.wire} TID: [0] [STRATOS] [2014-05-08 10:44:07,965] DEBUG {jclouds.wire} - >> "{}" {jclouds.wire} TID: [0] [STRATOS] [2014-05-08 10:44:07,982] DEBUG {jclouds.wire} - << "{"itemNotFound": {"message": "FloatingIpPoolNotFound: Floating ip pool not found.", "code": 404}}" {jclouds.wire} TID: [0] [STRATOS] [2014-05-08 10:44:07,987] ERROR {jclouds.compute} - createNodesInGroup(cisco-samp), completed: 0/1, errors: 1, rate: 5816ms/op {jclouds.compute} java.util.concurrent.ExecutionException: java.lang.NullPointerException at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:125) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.NullPointerException at org.jclouds.openstack.nova.v2_0.compute.functions.AllocateAndAddFloatingIpToNode.apply(AllocateAndAddFloatingIpToNode.java:100) at org.jclouds.openstack.nova.v2_0.compute.functions.AllocateAndAddFloatingIpToNode.apply(AllocateAndAddFloatingIpToNode.java:53) at com.google.common.util.concurrent.Futures$1.apply(Futures.java:711) at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:849) ... 3 more TID: [0] [STRATOS] [2014-05-08 10:44:07,988] ERROR {jclouds.compute} - createNodesInGroup(cisco-samp), completed: 0/1, errors: 1, rate: 5817ms/op {jclouds.compute} java.lang.RuntimeException: createNodesInGroup(cisco-samp), completed: 0/1, errors: 1, rate: 5817ms/op at org.jclouds.concurrent.FutureIterables.awaitCompletion(FutureIterables.java:155) at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:219) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) at com.sun.proxy.$Proxy91.createNodesInGroup(Unknown Source) at org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl.startInstance(CloudControllerServiceImpl.java:356) -- Best Regards, Nirmal Nirmal Fernando. PPMC Member & Committer of Apache Stratos, Senior Software Engineer, WSO2 Inc. Blog: http://nirmalfdo.blogspot.com/ -- Thanks & regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ [http://c.content.wso2.com/signatures/general.png]
