My feeling is that it is a race condition inside udev. There is a lot of
multipath devices on the system and I'm sometimes seeing messages that
udev is reaching its max children limit of 40.
The multipath -l <wwid> call issues a lot of change events to udev.
OpenStack calls it and immediately begins deleting the scsi devices that
it gets in the reply. Udev processes the messages in a multithreaded way
and the change event arrives after the delete event. It is possible?
OpenStack does this and the device sdbc is the one affected by the bug:
016-12-05 10:38:40.646 500 INFO nova.compute.manager
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b82c - - -]
[instance: a25caee0-2c74-4f91-a87b-e872cbd237e5] Detach volume
2992050a-d40d-4a6a-baa9-1a8f8624c838 from mountpoint /dev/vdb
2016-12-05 10:38:41.008 500 DEBUG oslo_concurrency.lockutils
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b82c
- - -] Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s
inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2016-12-05 10:38:41.009 500 DEBUG oslo_concurrency.processutils
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b
82c - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
multipath -l 360002ac00000000000006b3800014ee8 execute
/usr/lib/python2.7/dist-packages/
oslo_concurrency/processutils.py:223
2016-12-05 10:38:41.333 500 DEBUG oslo_concurrency.processutils
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b
82c - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
360002ac00000000000006b3800014ee8" returned: 0 in 0.324s execute
/usr/lib/python2.7/dist-package
s/oslo_concurrency/processutils.py:254
2016-12-05 10:38:41.334 500 DEBUG nova.storage.linuxscsi
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b82c - -
-] Found multipath device = /dev/mapper/360002ac00000000000006b3800014ee8
find_multipath_device
/usr/lib/python2.7/dist-packages/nova/storage/linuxscsi.py:135
2016-12-05 10:38:41.334 500 DEBUG nova.virt.libvirt.volume
[req-f5e6d9ff-1a0e-4455-8f60-0f9baea9d413 35c0bc8aaf3e43cea3f265b89c1216ee
e18099f000534ae89b8a978ec8e9b82c -
- -] devices to remove = [{'device': '/dev/sdbc', 'host': '1', 'id': '1',
'channel': '0', 'lun': '11'}, {'device': '/dev/sdbb', 'host': '1', 'id': '0',
'channel': '0',
'lun': '11'}, {'device': '/dev/sdbg', 'host': '2', 'id': '1', 'channel': '0',
'lun': '11'}, {'device': '/dev/sdbd', 'host': '2', 'id': '0', 'channel': '0',
'lun': '11'
}] disconnect_volume
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py:1403
2016-12-05 10:38:41.335 500 DEBUG nova.storage.linuxscsi [-] Trying (1) to
remove device /dev/sdbc _wait_for_remove
/usr/lib/python2.7/dist-packages/nova/storage/linuxs
csi.py:77
2016-12-05 10:38:41.335 500 DEBUG oslo_concurrency.processutils [-] Running cmd
(subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf tee -a
/sys/bus/scsi/drivers/sd
/1:0:1:11/delete execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:223
2016-12-05 10:38:41.455 500 DEBUG oslo_concurrency.processutils [-] CMD "sudo
nova-rootwrap /etc/nova/rootwrap.conf tee -a
/sys/bus/scsi/drivers/sd/1:0:1:11/delete" ret
urned: 0 in 0.119s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:254
2016-12-05 10:38:41.456 500 DEBUG oslo_concurrency.processutils [-] Running cmd
(subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf sginfo -r execute
/usr/lib/pyth
on2.7/dist-packages/oslo_concurrency/processutils.py:223
2016-12-05 10:38:41.543 500 DEBUG oslo_concurrency.processutils [-] CMD "sudo
nova-rootwrap /etc/nova/rootwrap.conf sginfo -r" returned: 0 in 0.087s execute
/usr/lib/py
thon2.7/dist-packages/oslo_concurrency/processutils.py:254
... and the same for the other 3 devices
Udev log, which is pretty unreadable because of all the multithreading, shows
this:
found 'b67:96' claiming
'/run/udev/links/\x2fdisk\x2fby-path\x2fpci-0000:05:00.0-fc-0x21120002ac014ee8-lun-11'
creating link '/dev/disk/by-path/pci-0000:05:00.0-fc-0x21120002ac014ee8-lun-11'
to '/dev/sdbc'
preserve already existing symlink
'/dev/disk/by-path/pci-0000:05:00.0-fc-0x21120002ac014ee8-lun-11' to
'../../sdbc'
...
handling device node '/dev/sdbc', devnum=b67:96, mode=0600, uid=0, gid=0
can not stat() node '/dev/sdbc' (No such file or directory)
created db file '/run/udev/data/b67:96' for
'/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-1/target1:0:1/1:0:1:11/block/sdbc'
adding watch on '/dev/sdbc'
inotify_add_watch(6, /dev/sdbc, 10) failed: No such file or directory
created db file '/run/udev/data/b67:96' for
'/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-1/target1:0:1/1:0:1:11/block/sdbc'
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1647067
Title:
Dangling UDEV links after removing FC LUNs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/udev/+bug/1647067/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
