Aled Sage created BROOKLYN-295:
----------------------------------
Summary: jclouds-openstack destroyNode failed after VM failed to
provision: id must be in format regionId/id
Key: BROOKLYN-295
URL: https://issues.apache.org/jira/browse/BROOKLYN-295
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Aled Sage
I attempted to deploy an app to OpenStack. It failed to provision due to
insufficient resources. But then jclouds tried and failed to destroy the VM
with the exception:
{noformat}
java.lang.IllegalArgumentException: id must be in format regionId/id
{noformat}
The provisioning error (shown in the OpenStack web-console) was:
{noformat}
Message No valid host was found. Exceeded max scheduling attempts 3 for
instance e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception: [u'Traceback
(most recent call last):\n', u' File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/c
Code 500
Details File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py",
line 669, in build_instances instances[0].uuid) File
"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py",
line 172, in populate_retry raise exception.NoValidHost(reason=msg)
Created June 8, 2016, 10:33 a.m.
{noformat}
The Brooklyn debug log shows:
{noformat}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation
[brooklyn-execmanager-yA1Lh5GF-504]: jclouds using template
{image={id=RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77,
providerId=e16a7bca-7363-45b1-bd6a-02479d4cea77, name=CentOS 7,
location={scope=REGION, id=RegionOne, description=RegionOne,
parent=openstack-nova}, os={family=centos, name=CentOS 7, version=,
description=CentOS 7, is64Bit=true}, description=CentOS 7, status=AVAILABLE,
loginUser=root}, hardware={id=RegionOne/2, providerId=2, name=m1.small,
location={scope=REGION, id=RegionOne, description=RegionOne,
parent=openstack-nova}, processors=[{cores=1.0, speed=1.0}], ram=2048,
volumes=[{type=LOCAL, size=20.0, bootDevice=true, durable=true}],
supportsImage=ALWAYS_TRUE}, location={scope=REGION, id=RegionOne,
description=RegionOne, parent=openstack-nova}, options={loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false}} / options {loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false} to provision machine in
openstack-nova:https:/acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation
[brooklyn-execmanager-yA1Lh5GF-504]: NOTE: unused flags passed to obtain VM in
openstack-nova:https://acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}:
{jclouds.keystone.credential-type=xxxxxxxx,
machineCreationSemaphore=java.util.concurrent.Semaphore@78d9e87b[Permits =
2147483643]}
2016-06-08 11:33:45,215 DEBUG jclouds.compute
[brooklyn-execmanager-yA1Lh5GF-504]: >> running 1 node
group(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29)
location(RegionOne) image(RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77)
hardwareProfile(RegionOne/2) options({loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true,
securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s,
brooklyn-user=aled, brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm,
brooklyn-entity-id=rj29jfrfvy, brooklyn-entity-name=swarm-node,
brooklyn-server-creation-date=2016-06-08-1133},
networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f], autoAssignFloatingIp=false,
keyPairName=openstack, configDrive=false})
2016-06-08 11:33:45,218 DEBUG o.j.l.s.f.RegionIdsFromConfiguration
[brooklyn-execmanager-yA1Lh5GF-504]: no jclouds.regions configured for provider
openstack-nova
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> GET
https://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a
HTTP/1.1
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> Accept:
application/json
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> X-Auth-Token:
50f7eaf091ea4ca494bbfc591d7f61a7
2016-06-08 12:03:48,855 DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService [user
thread 8]: Receiving response -2055078403: HTTP/1.1 200 OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << HTTP/1.1 200
OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Date: Wed, 08
Jun 2016 11:03:48 GMT
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Connection:
close
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: <<
X-Compute-Request-Id: req-8c17bd92-eada-4d90-b002-4cd2ef697343
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Content-Type:
application/json
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: <<
Content-Length: 2709
2016-06-08 12:03:48,855 DEBUG jclouds.wire [user thread 8]: << "{"server":
{"status": "ERROR", "updated": "2016-06-08T10:33:51Z", "hostId":
"8a47d0589f92964343fa21a82dfdc01abf880c069ffaf88c7bf8ba4a",
"OS-EXT-SRV-ATTR:host": "ds0017", "addresses": {}, "links": [{"href":
"http://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a";,
"rel": "self"}, {"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495
-4148-9a09-c4dee0d75e7a", "rel": "bookmark"}], "key_name": "openstack",
"image": {"id": "e16a7bca-7363-45b1-bd6a-02479d4cea77", "links": [{"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/images/e16a7bca-7363-45b1-bd6a-02479
d4cea77", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null,
"OS-EXT-STS:vm_state": "error", "OS-EXT-SRV-ATTR:instance_name":
"instance-00000d8b", "OS-SRV-USG:launched_at": null,
"OS-EXT-SRV-ATTR:hypervisor_hostname": "ds0017.sng01.blueboxgrid.com",
"flavor": {"id": "
2", "links": [{"href":
"http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/flavors/2";, "rel":
"bookmark"}]}, "id": "e74952b7-f495-4148-9a09-c4dee0d75e7a",
"OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_i
d": "111195567c864830ae4ade124a4a1efe", "name":
"brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823", "created":
"2016-06-08T10:33:48Z", "tenant_id": "691aa6dd3e4e482aaaf7f73b16fffc18",
"OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "
accessIPv4": "", "accessIPv6": "", "fault": {"message": "No valid host was
found. Exceeded max scheduling attempts 3 for instance
e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception: [u'Traceback (most recent
call last):\\n', u' File \"/opt/bbc/openstack-11.0-bbc173/n
ova/local/lib/python2.7/site-packages/nova/c", "code": 500, "details": " File
\"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py\",
line 669, in build_instances\n instances[0].uuid)\n File
\"/opt/bbc/openstack-11.0-bbc
173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 172,
in populate_retry\n raise exception.NoValidHost(reason=msg)\n", "created":
"2016-06-08T10:33:51Z"}, "OS-EXT-STS:power_state": 0, "config_drive": "",
"metadata": {"brooklyn-app-name": "Doc
ker Swarm", "brooklyn-entity-id": "rj29jfrfvy", "brooklyn-app-id":
"yfxfed97j6", "brooklyn-user": "aled", "brooklyn-entity-name": "swarm-node",
"brooklyn-server-creation-date": "2016-06-08-1133", "Name":
"brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s", "jc
louds-group": "brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29"}}}"
2016-06-08 12:03:48,860 WARN jclouds.compute [user thread 8]: Server
brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823 was not created
within 30 * 60ms so it will be destroyed.
2016-06-08 12:03:48,940 ERROR jclouds.compute [user thread 14]:
createNodesInGroup(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29),
completed: 0/1, errors: 1, rate: 1802112ms/op
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: id
must be in format regionId/id
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-16.0.1.jar:na]
at
org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123)
~[jclouds-core-1.9.2.jar:1.9.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: id must be in format regionId/id
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:125)
~[guava-16.0.1.jar:na]
at
org.jclouds.openstack.nova.v2_0.domain.regionscoped.RegionAndId.fromSlashEncoded(RegionAndId.java:31)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.functions.CleanupServer.apply(CleanupServer.java:66)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.destroyNode(NovaComputeServiceAdapter.java:269)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.createNodeWithGroupEncodedIntoName(NovaComputeServiceAdapter.java:152)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63)
~[jclouds-compute-1.9.2.jar:1.9.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_71]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:161)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
~[jclouds-compute-1.9.2.jar:1.9.2]
at
org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:154)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
at
org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210)
~[jclouds-compute-1.9.2.jar:1.9.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_71]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_71]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_71]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[guice-3.0.jar:na]
at com.sun.proxy.$Proxy115.createNodesInGroup(Unknown Source) ~[na:na]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.obtainOnce(JcloudsLocation.java:732)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.location.jclouds.JcloudsLocation.obtain(JcloudsLocation.java:622)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:406)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:396)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.Tasks.withBlockingDetails(Tasks.java:98)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:380)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:364)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at
org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
... 3 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
