** Also affects: linux (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1842751
Title:
[disco] [eoan] After unmount, cannot open /dev/vdb: Device or resource
busy
Status in OpenStack cinder charm:
New
Status in OpenStack nova-compute charm:
New
Status in linux package in Ubuntu:
Incomplete
Bug description:
Disco and Eoan device is busy after unmounting an ephemeral disk,
cannot format the device until rebooting.
This is blocking all of OpenStack Charms which interact with block
devices (Nova Compute, Ceph, Swift, Cinder), on the Disco and Eoan
series. As we are nearing LTS-1, this will become urgent pretty
quickly.
Reproducer, on an OpenStack cloud:
juju deploy cs:ubuntu ubuntu-bionic --series bionic
juju deploy cs:ubuntu ubuntu-disco --series disco
juju deploy cs:ubuntu ubuntu-eoan --series eoan
#### Succeeds on Bionic:
ubuntu@juju-8d01b7-foo-14:~$ uname -a
Linux juju-8d01b7-foo-14 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@juju-8d01b7-foo-14:~$ lsb_release -c
Codename: bionic
ubuntu@juju-8d01b7-foo-14:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk /mnt
ubuntu@juju-8d01b7-foo-14:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 985M 0 985M 0% /dev
tmpfs 200M 712K 199M 1% /run
/dev/vda1 20G 1.7G 18G 9% /
tmpfs 997M 0 997M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 997M 0 997M 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/vdb 40G 49M 38G 1% /mnt
tmpfs 100K 0 100K 0% /var/lib/lxd/shmounts
tmpfs 100K 0 100K 0% /var/lib/lxd/devlxd
tmpfs 200M 0 200M 0% /run/user/1000
ubuntu@juju-8d01b7-foo-14:~$ sudo umount /dev/vdb
ubuntu@juju-8d01b7-foo-14:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
meta-data=/dev/vdb isize=1024 agcount=4, agsize=2621440 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0,
reflink=0
data = bsize=4096 blocks=10485760, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=5120, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
#### Fails on Disco:
ubuntu@juju-8d01b7-foo-12:~$ uname -a
Linux juju-8d01b7-foo-12 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@juju-8d01b7-foo-12:~$ lsb_release -c
Codename: disco
ubuntu@juju-8d01b7-foo-12:~$ sudo umount /mnt
ubuntu@juju-8d01b7-foo-12:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@juju-8d01b7-foo-12:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
#### Fails on Eoan:
ubuntu@juju-8d01b7-foo-13:~$ sudo umount /mnt
ubuntu@juju-8d01b7-foo-13:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@juju-8d01b7-foo-13:~$ sudo mkfs.xfs -f -i size=1024 /dev/vdb
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
..
Original bug description:
On disco-stein, hook failed: "config-changed" with mkfs.xfs: cannot
open /dev/vdb: Device or resource busy
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/index.html
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-status-
zaza-5b39f0208674.txt
https://openstack-ci-
reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack
/charm-cinder/678676/3/3803/test_charm_func_full_7062/juju-crashdump-
7d3902a2-4fdf-435a-82c4-5d2ad9af4cb5.tar.xz
2019-09-03 21:33:27 DEBUG config-changed Unpacking apparmor-utils
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-libapparmor
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-apparmor
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up apparmor-utils
(2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Processing triggers for man-db
(2.8.5-2) ...
2019-09-03 21:33:30 INFO juju-log Setting up the apparmor profile for
usr.bin.nova-compute in enforce mode.
2019-09-03 21:33:30 DEBUG config-changed Setting
/etc/apparmor.d/usr.bin.nova-compute to enforce mode.
2019-09-03 21:33:30 DEBUG config-changed mkfs.xfs: cannot open /dev/vdb:
Device or resource busy
2019-09-03 21:33:30 DEBUG config-changed Traceback (most recent call last):
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
751, in <module>
2019-09-03 21:33:30 DEBUG config-changed main()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
744, in main
2019-09-03 21:33:30 DEBUG config-changed hooks.execute(sys.argv)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/hookenv.py",
line 914, in execute
2019-09-03 21:33:30 DEBUG config-changed self._hooks[hook_name]()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py",
line 1560, in wrapped_f
2019-09-03 21:33:30 DEBUG config-changed stopstart, restart_functions)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/host.py",
line 741, in restart_on_change_helper
2019-09-03 21:33:30 DEBUG config-changed r = lambda_f()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py",
line 1559, in <lambda>
2019-09-03 21:33:30 DEBUG config-changed (lambda: f(*args, **kwargs)),
__restart_map_cache['cache'],
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/hardening/harden.py",
line 93, in _harden_inner2
2019-09-03 21:33:30 DEBUG config-changed return f(*args, **kwargs)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/config-changed", line
264, in config_changed
2019-09-03 21:33:30 DEBUG config-changed
configure_local_ephemeral_storage()
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/nova_compute_utils.py",
line 1015, in configure_local_ephemeral_storage
2019-09-03 21:33:30 DEBUG config-changed mkfs_xfs(dev, force=True)
2019-09-03 21:33:30 DEBUG config-changed File
"/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/storage/linux/utils.py",
line 128, in mkfs_xfs
2019-09-03 21:33:30 DEBUG config-changed check_call(cmd)
2019-09-03 21:33:30 DEBUG config-changed File
"/usr/lib/python3.7/subprocess.py", line 347, in check_call
2019-09-03 21:33:30 DEBUG config-changed raise
CalledProcessError(retcode, cmd)
2019-09-03 21:33:30 DEBUG config-changed subprocess.CalledProcessError:
Command '['mkfs.xfs', '-f', '-i', 'size=1024', '/dev/vdb']' returned non-zero
exit status 1.
2019-09-03 21:33:30 ERROR juju.worker.uniter.operation runhook.go:132 hook
"config-changed" failed: exit status 1
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-cinder/+bug/1842751/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
