[Bug 1832384] Re: Unable to unmount apparently unused filesystem
@Colin this was indeed fixed by the backport of 4ecd55ea074217473f94cfee21bb72864d39f8d7, as we suspected it would be. I think this can be closed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We are going to set a system with a 4.15.0-60 kernel from the -proposed repository to confirm that this fixes the issue. Colin, do you know when that kernel will make its way out of -proposed and become generally available? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
It looks like items are in rare cases being removed from the export cache's hash table without being removed from the queue. Once they are removed from the hash table, no downcall can complete the request, because the item can't be looked up in the hash table, leaving the item orphaned, along with its reference to the filesystem we are trying to unmount. It looks like this has already been fixed upstream in https://github.com/torvalds/linux/commit/4ecd55ea074217473f94cfee21bb72864d39f8d7, and that commit has been pulled into an upcoming Ubuntu kernel release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We used BPF/BCC to trace all refcount changes on all of the relevant mounts, and for each (struct mount*, stack) pair, kept track of the number of references acquired/released by that pair. Including the struct mount* in the key in our map make the map quite large, but allowed us to pare the output down to a manageable size when we hit this issue by filtering out all of the refcount changes except those related to the one affected filesystem. Looking at all the reference count changes for a particular filesystem affected by this issue, we can see that the extra reference which is acquired but never released is being taken by code related to the NFSD export cache. In particular, the cache maintains a queue of requests which rpc.mountd is supposed to read and process. After processing a request, rpc.mountd is supposed to issue a downcall with the info requested by the kernel, at which point the item can be removed from the queue, and can be flushed when it expires, releasing the reference the item holds on the mount. rpc.mountd is waiting for requests, as it should be, but the kernel is not notifying mountd that there are requests in the queue. We have more investigation to do to understand how the kernel got into this state. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We are still seeing this issue and still working on tracking it down. We've done some tracing to track all of the refcount changes by stack when this occurs. However, the refcount is manipulated from many different contexts, making it difficult to identify which increment is the odd one out. We are currently working on a fix / work-around for another bug (https://www.spinics.net/lists/linux-nfs/msg74131.html) that is making our testing of this issue more difficult. Once we have that resolved, it should be a bit easier to make progress on this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
Thanks for taking a look. One of the things that is painful about debugging this issue is how long it takes to manifest. It would probably require several weeks of testing a particular kernel version without seeing the issue before we could be reasonably sure that the problem is fixed. Also, testing in parallel can be quite resource intensive since we need 10-15 machines to drive the workload for a single machine under test. Given this, I think we will spend a couple more days trying to come up with a better reproducer, one that can produce the issue in less time, or at least without requiring our application (and therefore our ZFS bits). If we can do that, we should be in a better position to test with other kernels. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] [NEW] Unable to unmount apparently unused filesystem
Public bug reported: We periodically see an issue where unmounting a ZFS filesystem fails with EBUSY, even though there appears to be no one using it. # cat /proc/self/mounts | grep /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive zfs rw,nosuid,nodev,noexec,relatime,xattr,noacl 0 0 'lsof' and 'fuser' show no processes using any of the files in the problematic filesystem: # ls -l /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive/ total 221 -rw-r- 1 500 500 52736 May 22 11:01 1_19_1008904362.dbf -rw-r- 1 500 500 541696 May 22 11:03 1_20_1008904362.dbf # fuser /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive/1_20_1008904362.dbf # fuser /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive/1_19_1008904362.dbf # fuser /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive/ # lsof | grep /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive # The filesystem was shared over NFS, but has since been unshared: # showmount -e | grep /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive # Since no one appears to be using the filesystem, our expectation is that it should be possible to unmount the filesystem. However, attempts to unmount the filesystem fail with EBUSY: # zfs destroy domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive umount: /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive: target is busy. cannot unmount '/domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive': umount failed # umount /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive umount: /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive: target is busy. Using bpftrace, we can see that the unmount is failing in 'propagate_mount_busy()' in the kernel. Using a live kernel debugger, we can look at the 'mount' struct for this particular mount and see that the 'mnt_count' refcount summed across all CPUs is 2. For filesystems that are eligible for unmounting, the refcount is 1. The only way to work around this issue that we have found is to reboot, at which point the filesystem can be unmounted and destroyed. So far, we have only been able to reproduce this using a workload driven by our application. The application mananges ZFS filesystems in groups, and the lifecycle of each group looks something like - Create and mount a group of filesystems, 1 parent and 4 children: /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370 /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/datafile /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/external /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/archive /domain0/group-38/oracle_db_container-202/oracle_timeflow-16370/temp - Share all 5 filesystems over NFS - A client mounts all 5 shares using NFSv3 - For a few hours, the client does NFS operations on the filesystems and the server occasionally takes ZFS snapshots of them - Unshare filesystems - Unmount filesystems - Delete filesystems These groups of filesystems are constantly being created and destroyed. At any given time, we have ~30k filesystems on the system, about 5k of which are shared. On average, one out of ~200-300k unmounts fails with this EBUSY error. To create and destroy this many filesystems takes us about a week or so. Note that we are using ZFS built from https://github.com/delphix/zfs, which is essentially master ZFS on Linux. ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: linux-image-4.15.0-50-generic 4.15.0-50.54 ProcVersionSignature: Ubuntu 4.15.0-50.54-generic 4.15.18 Uname: Linux 4.15.0-50-generic x86_64 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 20 19:10 seq crw-rw 1 root audio 116, 33 May 20 19:10 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.6 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Tue Jun 11 05:28:21 2019 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb: Error: [Errno 2] No such file or directory: 'lsusb': 'lsusb' MachineType: VMware, Inc. VMware Virtual Platform PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 svgadrmfb ProcKernelCmdLine:
[Bug 1661629] Re: upgrade of kernel fails with mkinitramfs: failed to determine device for /
@Mathieu, thanks for your work on this. Regarding the test cases, I think that the second case needs an additional step: -- initramfs update -- 1) install Ubuntu, using ZFS as a rootfs. 2) change 'MODULES=most' to 'MODULES=dep' in /etc/initramfs-tools/initramfs.conf 3) run "sudo update-initramfs -u" Without the second step, 'update-initramfs -u' will succeed even without the fix because 'MODULES=most' will probably result in the zfs module and necessary drivers being included anyway. I ran through this updated version of the test case on 18.04 with the package from the -proposed repository, and verified that the new version of the package fixed the issue. As far as the first test case, I'm not sure I have a simple way to test that one. I have a straightforward way of building with zfs on root for 18.04 and newer, but not 16.04. However, looking at the original bug report, it looks like the issue was building the initramfs for the kdump kernel, not the main kernel. I suspect that test case would need to include installing kdump-tools. This is for the same reason that test case #1 needs to be modified: by default, the initramfs for the main kernel is built with MODULES=most, whereas the kdump kernel by default is built with MODULES=dep, and this bug only affects making an initramfs with MODULES=dep. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1661629 Title: upgrade of kernel fails with mkinitramfs: failed to determine device for / To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1661629] Re: upgrade of kernel fails with mkinitramfs: failed to determine device for /
** Patch added: "mkinitramfs.patch" https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+attachment/5162606/+files/mkinitramfs.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1661629 Title: upgrade of kernel fails with mkinitramfs: failed to determine device for / To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1661629] Re: upgrade of kernel fails with mkinitramfs: failed to determine device for /
> It might be easier to resolve this by considering this as "mkinitramfs
fails on ZFS root without /etc/fstab lines for the ZFS filesystems when
MODULES=dep is used".
Yes, that sounds like an accurate summary. I'm not sure I follow how
/etc/fstab is related, though. Looking briefly through mkinitramfs, I
don't see where that is being used.
> Could you perhaps see if you can remove kdump-tools from the equation
like this?
Good idea. Here is a more minimal way to reproduce the issue: with ZFS on root
- set 'MODULES=dep' in '/etc/initramfs-tools/initramfs.conf'
- run 'sudo update-initramfs -u'
Below is an updated patch, which finds and adds the module for each
device in the zfs pool containing the root filesystem. I've tested the
patch on Bionic and Cosmic with both zfs and ext4 root filesystems. Note
that I wasn't entirely sure if we need to handle the case where
'dev_node' is /dev/root. I'm not familiar with that setup, so I'm not
sure whether that could even apply when using zfs on root.
--- hook-functions 2018-07-10 22:19:18.489142772 +
+++ /usr/share/initramfs-tools/hook-functions 2018-07-10 22:06:43.969661842
+
@@ -357,6 +357,21 @@
return
fi
+ if [ "${FSTYPE}" = "zfs" ]; then
+ manual_add_modules "${FSTYPE}"
+
+ # ZFS uses the name of a filesystem instead of a device. Find
+ # the devices that make up the pool containing the specified
+ # filesystem, and add the appropriate driver for each device.
+ local poolname="${dev_node%%/*}"
+ zpool list -vPL "$poolname" | while read dev ignored; do
+ # Ignore non-leaf vdevs by skipping anything that
doesn't
+ # look like an absolute path
+ echo "$dev" | grep -q '^/' && block_dev_mod_add "$dev"
+ done
+ return
+ fi
+
if [ "$dir" = / ] && [ "${dev_node}" = "/dev/root" ] ; then
if [ -b "${dev_node}" ]; then
# Match it to the canonical device name by UUID
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1661629
Title:
upgrade of kernel fails with mkinitramfs: failed to determine device
for /
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1661629] Re: upgrade of kernel fails with mkinitramfs: failed to determine device for /
> I'm not particularly familiar with this part of initramfs-tools, but it seems to me that essentials like the loading of the virtio_pci module would be short-circuited by your patch. Yes, you're right, thanks for pointing that out. If the driver isn't built-in to the kernel, it would be omitted. I think what we need to do here is get a list of the devices that make up the root pool, probably from the output of `zpool list -vPL`, and then call 'block_dev_mod_add' on each device. I'll work on updating the patch. > I was just pointed to https://github.com/zfsonlinux/zfs/wiki/Ubuntu-18.04-Root-on-ZFS and told that ZFS filesystems aren't expected to be in /etc/fstab according to those instructions. Is this correct? I haven't checked but it seems that initramfs-tools wouldn't then fail under these conditions. It's true that we don't have the filesystems listed in /etc/fstab. Can you elaborate on how you anticipate that would affect building the initramfs? > Could you provide exact steps to reproduce your problem, please? Once you have a machine with zfs on root, this should be reproducible by running `apt install kdump-tools`. The post install hooks fail while trying to build an initramfs for the crash kernel. We build these ubuntu images that use zfs on root using live-build (https://github.com/delphix/appliance-build/), but our process currently relies on some packages internal to our organization. If need be, we might be able to modify the build to be usable externally as well. > Does this apply and work correctly on the current Ubuntu development release > (Cosmic) please? We land changes there first and only once successful > consider backporting to existing releases as appropriate. Is it possible to upgrade my machine to the development release? If not, I might be able to modify our build process to build an Cosmic image that uses zfs on root. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1661629 Title: upgrade of kernel fails with mkinitramfs: failed to determine device for / To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1661629] Re: upgrade of kernel fails with mkinitramfs: failed to determine device for /
Maybe there is something better, but it seems like the simplest change
that would fix this would be something along the lines of the patch
below. This just assumes that if the filesystem type is listed as zfs
then it actually is zfs, and it adds the zfs module. With this change I
was able to build a image using "MODULES=dep" and successfully create a
crash dump with zfs on root.
--- /usr/share/initramfs-tools/hook-functions.orig 2018-06-26
22:24:55.918498180 +
+++ /usr/share/initramfs-tools/hook-functions 2018-06-26 22:48:44.350437446
+
@@ -350,9 +350,12 @@
return
fi
- # handle ubifs and return since ubifs is mounted on char devices
- # but most of the commands below only work with block devices.
- if [ "${FSTYPE}" = "ubifs" ]; then
+ # Most of the commands below only work with block devices. Some
+ # file systems don't specify a block device:
+ # - ubifs is mounted on char devices
+ # - zfs specifies the name of a zfs dataset
+ # In these cases, just assume the listed fs type is correct.
+ if [ "${FSTYPE}" = "ubifs" -o "${FSTYPE}" = "zfs" ]; then
manual_add_modules "${FSTYPE}"
return
fi
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1661629
Title:
upgrade of kernel fails with mkinitramfs: failed to determine device
for /
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1661629/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
