Public bug reported:
Problem description:
The nova interface-attach command removes pre-existing neutron ports from the
environment if it fails to attach to an instance _even_ where '--port-id' has
been specified. This behaviour was introduced by fixing bug #1338551 [1].
Steps to reproduce:
1) create a new neutron port
$ neutron port-create --name <port-name> <network-name>
2) boot an instance (make sure to specify a keypair and check sec groups for
ssh connectivity to the instance)
$ nova boot ...
3) [OPTIONAL] add/remove the port several times over to prove the funtionality
is working OK.
$ nova interface-attach --port-id <port-id> <instance-name>
$ nova interface-detach <instance-name> <port-id>
4) simulate a kernel crash on the instance, as this should cause a scenario
where an interface attach will fail (ssh connectivity is assumed for this step)
$ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c >
/proc/sysrq-trigger' while connected to the instance
4a. Verify the kernel has actually crashed
$ nova console-log <instance-name>
5) try to attach the port while the instance is still crashed # **note** if the
port hasn't been attached before (i.e. you skipped step 3, it may succeed
initially then it will fail on subsequent attach attempts). Also, at this point
it should not matter if the port is still attached to the instance.
$ nova interface-attach --port-id <port-id> <instance-name>
Errors observed:
$ nova interface-attach --port-id <port-id> <instance-name>
ERROR: Failed to attach interface (HTTP 500) (Request-ID:
req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
Expected results:
The port should still exist after failure in this scenario.
Actual results:
'neutron port-list' will no longer show the port. It has been removed.
The port is removed from the environment and therefore is no longer available.
Snippet from /var/log/nova/nova-compute.log
[instance: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] attaching network adapter
failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
1263, in attach_interface
virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in
attachDeviceFlags
if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',
dom=self)
libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource
busy
attach interface failed , try to deallocate port
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter
device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Exception during message handling: Failed to attach network adapter device to
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Full error:
http://pastebin.ubuntu.com/15471511/
$ sudo apt-cache policy nova-compute
nova-compute:
Installed: 1:2015.1.2-0ubuntu2~cloud0
Ubuntu 14.04.4 LTS
Why does this matter:
As specified in [1], where a port has been attached using --net-id option it is
automatically created before attaching to the VM. Therefore, it is the correct
behaviour to cleanup after a failure to attach.
Where "--port-id" has been specified, it should not be assumed that it was auto
created, it has been specifically created and therefore may have pre-existed
the VM, this means the port should be re-usable if desired and therefore should
not be cleaned up in the case of attach failure. When the port has been
pre-created and '--port-id' is specified in the interface-attach command, if
the action fails to attach it should be handled without being removed from the
environment and exist for re-assignment to another instance or for retry to the
original instance once it has recovered from it's failure.
This behaviour is confirmed on both Kilo and Liberty.
Related bugs:
[1] https://bugs.launchpad.net/nova/+bug/1338551
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
Problem description:
The nova interface-attach command removes pre-existing neutron ports from the
environment if it fails to attach to an instance _even_ where '--port-id' has
been specified. This behaviour was introduced by fixing bug #1338551 [1].
Steps to reproduce:
1) create a new neutron port
- $ neutron port-create --name <port-name> <network-name>
+ $ neutron port-create --name <port-name> <network-name>
2) boot an instance (make sure to specify a keypair and check sec groups for
ssh connectivity to the instance)
- $ nova boot ...
+ $ nova boot ...
3) [OPTIONAL] add/remove the port several times over to prove the
funtionality is working OK.
- $ nova interface-attach --port-id <port-id> <instance-name>
- $ nova interface-detach <instance-name> <port-id>
+ $ nova interface-attach --port-id <port-id> <instance-name>
+ $ nova interface-detach <instance-name> <port-id>
4) simulate a kernel crash on the instance, as this should cause a scenario
where an interface attach will fail (ssh connectivity is assumed for this step)
- $ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c >
/proc/sysrq-trigger' while connected to the instance
- 4a. Verify the kernel has actually crashed
- $ nova console-log <instance-name>
+ $ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c >
/proc/sysrq-trigger' while connected to the instance
+ 4a. Verify the kernel has actually crashed
+ $ nova console-log <instance-name>
5) try to attach the port while the instance is still crashed # **note** if
the port hasn't been attached before (i.e. you skipped step 3, it may succeed
initially then it will fail on subsequent attach attempts). Also, at this point
it should not matter if the port is still attached to the instance.
- $ nova interface-attach --port-id <port-id> <instance-name>
+ $ nova interface-attach --port-id <port-id> <instance-name>
Errors observed:
$ nova interface-attach --port-id <port-id> <instance-name>
ERROR: Failed to attach interface (HTTP 500) (Request-ID:
req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
Expected results:
The port should still exist after failure in this scenario.
Actual results:
'neutron port-list' will no longer show the port. It has been removed.
The port is removed from the environment and therefore is no longer available.
- Snippet from /var/log/nova/nova.log
+ Snippet from /var/log/nova/nova-compute.log
[instance: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] attaching network adapter
failed.
- Traceback (most recent call last):
- File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
1263, in attach_interface
- virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
- File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in
doit
- result = proxy_call(self._autowrap, f, *args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
- rv = execute(f, *args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
execute
- six.reraise(c, e, tb)
- File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
tworker
- rv = meth(*args, **kwargs)
- File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in
attachDeviceFlags
- if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',
dom=self)
- libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource
busy
- attach interface failed , try to deallocate port
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter
device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- Exception during message handling: Failed to attach network adapter device
to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+ Traceback (most recent call last):
+ File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
1263, in attach_interface
+ virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
+ File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in
doit
+ result = proxy_call(self._autowrap, f, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
+ rv = execute(f, *args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
execute
+ six.reraise(c, e, tb)
+ File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
tworker
+ rv = meth(*args, **kwargs)
+ File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in
attachDeviceFlags
+ if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',
dom=self)
+ libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource
busy
+ attach interface failed , try to deallocate port
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter
device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+ Exception during message handling: Failed to attach network adapter device
to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Full error:
http://pastebin.ubuntu.com/15471511/
$ sudo apt-cache policy nova-compute
nova-compute:
- Installed: 1:2015.1.2-0ubuntu2~cloud0
+ Installed: 1:2015.1.2-0ubuntu2~cloud0
Ubuntu 14.04.4 LTS
-
Why does this matter:
As specified in [1], where a port has been attached using --net-id option it
is automatically created before attaching to the VM. Therefore, it is the
correct behaviour to cleanup after a failure to attach.
Where "--port-id" has been specified, it should not be assumed that it was
auto created, it has been specifically created and therefore may have
pre-existed the VM, this means the port should be re-usable if desired and
therefore should not be cleaned up in the case of attach failure. When the port
has been pre-created and '--port-id' is specified in the interface-attach
command, if the action fails to attach it should be handled without being
removed from the environment and exist for re-assignment to another instance or
for retry to the original instance once it has recovered from it's failure.
This behaviour is confirmed on both Kilo and Liberty.
Related bugs:
[1] https://bugs.launchpad.net/nova/+bug/1338551
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1560472
Title:
nova interface-attach command removes pre-existing neutron ports from
the environment if it fails to attach to an instance _even_ where
'--port-id' has been specified
Status in OpenStack Compute (nova):
New
Bug description:
Problem description:
The nova interface-attach command removes pre-existing neutron ports from the
environment if it fails to attach to an instance _even_ where '--port-id' has
been specified. This behaviour was introduced by fixing bug #1338551 [1].
Steps to reproduce:
1) create a new neutron port
$ neutron port-create --name <port-name> <network-name>
2) boot an instance (make sure to specify a keypair and check sec groups for
ssh connectivity to the instance)
$ nova boot ...
3) [OPTIONAL] add/remove the port several times over to prove the
funtionality is working OK.
$ nova interface-attach --port-id <port-id> <instance-name>
$ nova interface-detach <instance-name> <port-id>
4) simulate a kernel crash on the instance, as this should cause a scenario
where an interface attach will fail (ssh connectivity is assumed for this step)
$ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c >
/proc/sysrq-trigger' while connected to the instance
4a. Verify the kernel has actually crashed
$ nova console-log <instance-name>
5) try to attach the port while the instance is still crashed # **note** if
the port hasn't been attached before (i.e. you skipped step 3, it may succeed
initially then it will fail on subsequent attach attempts). Also, at this point
it should not matter if the port is still attached to the instance.
$ nova interface-attach --port-id <port-id> <instance-name>
Errors observed:
$ nova interface-attach --port-id <port-id> <instance-name>
ERROR: Failed to attach interface (HTTP 500) (Request-ID:
req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
Expected results:
The port should still exist after failure in this scenario.
Actual results:
'neutron port-list' will no longer show the port. It has been removed.
The port is removed from the environment and therefore is no longer available.
Snippet from /var/log/nova/nova-compute.log
[instance: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] attaching network adapter
failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
1263, in attach_interface
virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in
attachDeviceFlags
if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed',
dom=self)
libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource
busy
attach interface failed , try to deallocate port
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter
device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Exception during message handling: Failed to attach network adapter device
to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Full error:
http://pastebin.ubuntu.com/15471511/
$ sudo apt-cache policy nova-compute
nova-compute:
Installed: 1:2015.1.2-0ubuntu2~cloud0
Ubuntu 14.04.4 LTS
Why does this matter:
As specified in [1], where a port has been attached using --net-id option it
is automatically created before attaching to the VM. Therefore, it is the
correct behaviour to cleanup after a failure to attach.
Where "--port-id" has been specified, it should not be assumed that it was
auto created, it has been specifically created and therefore may have
pre-existed the VM, this means the port should be re-usable if desired and
therefore should not be cleaned up in the case of attach failure. When the port
has been pre-created and '--port-id' is specified in the interface-attach
command, if the action fails to attach it should be handled without being
removed from the environment and exist for re-assignment to another instance or
for retry to the original instance once it has recovered from it's failure.
This behaviour is confirmed on both Kilo and Liberty.
Related bugs:
[1] https://bugs.launchpad.net/nova/+bug/1338551
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1560472/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
