Aled Sage created JCLOUDS-1283:
----------------------------------
Summary: IllegalArgumentException in OpenStack destroyNode, after
failure to provision
Key: JCLOUDS-1283
URL: https://issues.apache.org/jira/browse/JCLOUDS-1283
Project: jclouds
Issue Type: Bug
Components: jclouds-compute
Affects Versions: 1.9.2
Reporter: Aled Sage
Priority: Trivial
As reported in https://issues.apache.org/jira/browse/BROOKLYN-295, with jclouds
1.9.2 (looking at the line numbers, it looks like 2.0.1 would have the same
behaviour).
I attempted to deploy an app to OpenStack. It failed to provision due to
insufficient resources (hence tricky to reproduce). But then jclouds tried and
failed to destroy the VM.
It would be nice if jclouds handled the failure more gracefully. This exception
hides the message about the provisioning request failing.
The exception in jclouds was:
{noformat}
java.lang.IllegalArgumentException: id must be in format regionId/id
{noformat}
The provisioning error (shown in the OpenStack web-console) was:
{noformat}
Message No valid host was found. Exceeded max scheduling attempts 3 for
instance e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception: [u'Traceback
(most recent call last):\n', u' File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/c
Code 500
Details File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py",
line 669, in build_instances instances[0].uuid) File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py",
line 172, in populate_retry raise exception.NoValidHost(reason=msg)
Created June 8, 2016, 10:33 a.m.
{noformat}
The Brooklyn debug log shows:
{noformat}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation
[brooklyn-execmanager-yA1Lh5GF-504]: jclouds using template
{image={id=RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77,
providerId=e16a7bca-7363-45b1-bd6a-02479d4cea77, name=CentOS 7,
location={scope=REGION, id=RegionOne, description=RegionOne,
parent=openstack-nova}, os={family=centos, name=CentOS 7, version=,
description=CentOS 7, is64Bit=true}, description=CentOS 7, status=AVAILABLE,
loginUser=root}, hardware={id=RegionOne/2, providerId=2, name=m1.small,
location={scope=REGION, id=RegionOne, description=RegionOne,
parent=openstack-nova}, processors=[{cores=1.0, speed=1.0}], ram=2048,
volumes=[{type=LOCAL, size=20.0, bootDevice=true, durable=true}],
supportsImage=ALWAYS_TRUE}, location={scope=REGION, id=RegionOne,
description=RegionOne, parent=openstack-nova}, options={loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false}} / options {loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false} to provision machine in
openstack-nova:https:/acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation
[brooklyn-execmanager-yA1Lh5GF-504]: NOTE: unused flags passed to obtain VM in
openstack-nova:https://acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}:
{jclouds.keystone.credential-type=xxxxxxxx,
machineCreationSemaphore=java.util.concurrent.Semaphore@78d9e87b[Permits =
2147483643]}
2016-06-08 11:33:45,215 DEBUG jclouds.compute
[brooklyn-execmanager-yA1Lh5GF-504]: >> running 1 node
group(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29)
location(RegionOne) image(RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77)
hardwareProfile(RegionOne/2) options({loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false})
2016-06-08 11:33:45,218 DEBUG o.j.l.s.f.RegionIdsFromConfiguration
[brooklyn-execmanager-yA1Lh5GF-504]: no jclouds.regions configured for provider
openstack-nova
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> GET
https://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a
HTTP/1.1
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> Accept:
application/json
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> X-Auth-Token:
50f7eaf091ea4ca494bbfc591d7f61a7
2016-06-08 12:03:48,855 DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService [user
thread 8]: Receiving response -2055078403: HTTP/1.1 200 OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << HTTP/1.1 200
OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Date: Wed, 08
Jun 2016 11:03:48 GMT
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Connection:
close
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: <<
X-Compute-Request-Id: req-8c17bd92-eada-4d90-b002-4cd2ef697343
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Content-Type:
application/json
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: <<
Content-Length: 2709
2016-06-08 12:03:48,855 DEBUG jclouds.wire [user thread 8]: << "{"server":
{"status": "ERROR", "updated": "2016-06-08T10:33:51Z", "hostId":
"8a47d0589f92964343fa21a82dfdc01abf880c069ffaf88c7bf8ba4a",
"OS-EXT-SRV-ATTR:host": "ds0017", "addresses": {}, "links": [{"href":
"http://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a";,
"rel": "self"}, {"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495
-4148-9a09-c4dee0d75e7a", "rel": "bookmark"}], "key_name": "openstack",
"image": {"id": "e16a7bca-7363-45b1-bd6a-02479d4cea77", "links": [{"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/images/e16a7bca-7363-45b1-bd6a-02479
d4cea77", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null,
"OS-EXT-STS:vm_state": "error", "OS-EXT-SRV-ATTR:instance_name":
"instance-00000d8b", "OS-SRV-USG:launched_at": null,
"OS-EXT-SRV-ATTR:hypervisor_hostname": "ds0017.sng01.blueboxgrid.com",
"flavor": {"id": "
2", "links": [{"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/flavors/2";, "rel":
"bookmark"}]}, "id": "e74952b7-f495-4148-9a09-c4dee0d75e7a",
"OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_i
d": "111195567c864830ae4ade124a4a1efe", "name":
"brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823", "created":
"2016-06-08T10:33:48Z", "tenant_id": "691aa6dd3e4e482aaaf7f73b16fffc18",
"OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "
accessIPv4": "", "accessIPv6": "", "fault": {"message": "No valid host was
found. Exceeded max scheduling attempts 3 for instance
e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception: [u'Traceback (most recent
call last):\\n', u' File \"/opt/bbc/openstack-11.0-bbc173/n
ova/local/lib/python2.7/site-packages/nova/c", "code": 500, "details": " File
\"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py\",
line 669, in build_instances\n instances[0].uuid)\n File
\"/opt/bbc/openstack-11.0-bbc
173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 172,
in populate_retry\n raise exception.NoValidHost(reason=msg)\n", "created":
"2016-06-08T10:33:51Z"}, "OS-EXT-STS:power_state": 0, "config_drive": "",
"metadata": {"brooklyn-app-name": "Doc
ker Swarm", "brooklyn-entity-id": "rj29jfrfvy", "brooklyn-app-id":
"yfxfed97j6", "brooklyn-user": "aled", "brooklyn-entity-name": "swarm-node",
"brooklyn-server-creation-date": "2016-06-08-1133", "Name":
"brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s", "jc
louds-group": "brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29"}}}"
2016-06-08 12:03:48,860 WARN jclouds.compute [user thread 8]: Server
brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823 was not created
within 30 * 60ms so it will be destroyed.
2016-06-08 12:03:48,940 ERROR jclouds.compute [user thread 14]:
createNodesInGroup(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29),
completed: 0/1, errors: 1, rate: 1802112ms/op
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: id
must be in format regionId/id
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-16.0.1.jar:na]
at
org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123)
~[jclouds-core-1.9.2.jar:1.9.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: id must be in format regionId/id
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:125)
~[guava-16.0.1.jar:na]
at
org.jclouds.openstack.nova.v2_0.domain.regionscoped.RegionAndId.fromSlashEncoded(RegionAndId.java:31)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.functions.CleanupServer.apply(CleanupServer.java:66)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.destroyNode(NovaComputeServiceAdapter.java:269)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.createNodeWithGroupEncodedIntoName(NovaComputeServiceAdapter.java:152)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63)
~[jclouds-compute-1.9.2.jar:1.9.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_71]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:161)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:154)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210)
~[jclouds-compute-1.9.2.jar:1.9.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_71]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_71]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_71]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[guice-3.0.jar:na]
at com.sun.proxy.$Proxy115.createNodesInGroup(Unknown Source) ~[na:na]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.obtainOnce(JcloudsLocation.java:732)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.obtain(JcloudsLocation.java:622)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:406)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:396)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.Tasks.withBlockingDetails(Tasks.java:98)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:380)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:364)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
... 3 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
