[Bug 1832384] Re: Unable to unmount apparently unused filesystem
** Changed in: linux (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
@Colin this was indeed fixed by the backport of 4ecd55ea074217473f94cfee21bb72864d39f8d7, as we suspected it would be. I think this can be closed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
@John, I was wondering what to do about this bug report. Is it still an issue or shall I close it? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
See also: https://wiki.ubuntu.com/Kernel/StableReleaseCadence and https://kernel.ubuntu.com/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We generally have a 3 week release cycle on kernels, so if it's in -proposed it probably in the later 1-2 weeks of this cycle. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
latter 1-2 weeks of this cycle -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We are going to set a system with a 4.15.0-60 kernel from the -proposed repository to confirm that this fixes the issue. Colin, do you know when that kernel will make its way out of -proposed and become generally available? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
That's really helpful to know John, thanks for the feedback. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
It looks like items are in rare cases being removed from the export cache's hash table without being removed from the queue. Once they are removed from the hash table, no downcall can complete the request, because the item can't be looked up in the hash table, leaving the item orphaned, along with its reference to the filesystem we are trying to unmount. It looks like this has already been fixed upstream in https://github.com/torvalds/linux/commit/4ecd55ea074217473f94cfee21bb72864d39f8d7, and that commit has been pulled into an upcoming Ubuntu kernel release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We used BPF/BCC to trace all refcount changes on all of the relevant mounts, and for each (struct mount*, stack) pair, kept track of the number of references acquired/released by that pair. Including the struct mount* in the key in our map make the map quite large, but allowed us to pare the output down to a manageable size when we hit this issue by filtering out all of the refcount changes except those related to the one affected filesystem. Looking at all the reference count changes for a particular filesystem affected by this issue, we can see that the extra reference which is acquired but never released is being taken by code related to the NFSD export cache. In particular, the cache maintains a queue of requests which rpc.mountd is supposed to read and process. After processing a request, rpc.mountd is supposed to issue a downcall with the info requested by the kernel, at which point the item can be removed from the queue, and can be flushed when it expires, releasing the reference the item holds on the mount. rpc.mountd is waiting for requests, as it should be, but the kernel is not notifying mountd that there are requests in the queue. We have more investigation to do to understand how the kernel got into this state. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
We are still seeing this issue and still working on tracking it down. We've done some tracing to track all of the refcount changes by stack when this occurs. However, the refcount is manipulated from many different contexts, making it difficult to identify which increment is the odd one out. We are currently working on a fix / work-around for another bug (https://www.spinics.net/lists/linux-nfs/msg74131.html) that is making our testing of this issue more difficult. Once we have that resolved, it should be a bit easier to make progress on this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
Any updates on this issue? ** Changed in: linux (Ubuntu) Status: In Progress => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
Thanks for taking a look. One of the things that is painful about debugging this issue is how long it takes to manifest. It would probably require several weeks of testing a particular kernel version without seeing the issue before we could be reasonably sure that the problem is fixed. Also, testing in parallel can be quite resource intensive since we need 10-15 machines to drive the workload for a single machine under test. Given this, I think we will spend a couple more days trying to come up with a better reproducer, one that can produce the issue in less time, or at least without requiring our application (and therefore our ZFS bits). If we can do that, we should be in a better position to test with other kernels. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
I had a quick look at there are quite a few differences between the Delphix ZFS git repo and the upstream ZFS git repo. Github states: "This branch is 376 commits ahead of zfsonlinux:master.". So: 1. Does this issue occur with stock ZFS rather than Delphix ZFS? 2. With the activity you are performing to trigger the bug it does seem like there is a race condition occurring on mnt_count so this does look like a kernel bug. One way to quickly sanity check this is with a coarse bisect using some pre-build ubuntu kernels based on the mainline kernels. The wiki page https://wiki.ubuntu.com/Kernel/MainlineBuilds details our mainline builds, you can select a mainline kernel from: https://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D - install it, run the tests and see if the problem is fixed or still persists. I suggest trying a recent (say 5.1) kernel build, and if that works, bisect on that between 4.15 and 5.1 etc until you find the earliest fix point. If the problem still occurs with a 5.1 kernel, then we need to start digging a bit deeper. But I think it is definitely worth trying a recent kernel first as the fix may be already be in an upstream kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1832384] Re: Unable to unmount apparently unused filesystem
** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: linux (Ubuntu) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1832384 Title: Unable to unmount apparently unused filesystem To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1832384/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
