[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
I tested the eoan kernel 5.2.0-10-generic and the suspected problems in the block layer didn't happen anymore after removing the workarounds (artificial delays and module insertion) needed by the bionic kernel 4.15.0-54-generic used in core18. 5.3.0-050300rc4-generic also works without any immediately visible problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
That sounds awesome, Claudio, thank you! We'll start with Eoan and I will work on getting a debug kernel ready in the meantime. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
Thanks Connor. My test configuration is a custom core system using the official core18 kernel snap on a KVM-based virtual machine. The kernel snap is installed from channel 18 (snap download --channel 18 pc-kernel) and my current revision is 240. That said, it's easy to test new kernels as long as I can convert them to a snap package. Foundations is currently preparing a kernel snap based on eoan so I can check with this newer kernel if the problems persist (it could be a first check for a bisect), but I could also boot an instrumented kernel that would give us a better insight on what could be happening. So, to summarize: we can use either approach, but my feeling is that the debug kernel and the eoan kernel could be a good start to help us to gather more data points. Ideally I could start testing that next week after returning from Toronto. Would that work for you? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
Hey Claudio, Did you experience this only after upgrading from a previous kernel? If so, what was the kernel that you upgraded from? I've been trying to reproduce this issue locally in a VM but haven't had any luck so far. So, I've been working from the kernel Oops in comment #1 and I am pretty sure I see where the NULL pointer is coming from but I am not sure yet where it's being to NULL. Since this smells a little bit like a race condition (unconfirmed) I'm trying to find ways to narrow down the search radius. I have a couple of options in mind for moving forward: 1. We can have you try out a few mainline kernels. If you experience the bug in one of the kernels but not another, we can "bisect" between mainline kernels to zero in on the culpable patch. I would be very interested to see if you experience the same regression on the latest mainline kernel, as that may indicate that the regression also exists in mainline. 2. I can continue reading through the code and prepare a debug kernel that will hopefully give us more information when the bug is triggered. My goal behind this would be to narrow down precisely where/how that NULL pointer is being set. Let me know if either of those options sound feasible to you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Changed in: linux (Ubuntu) Assignee: Connor Kuehl (connork) => (unassigned) ** Changed in: linux (Ubuntu) Importance: Medium => Undecided -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
Thanks. I tested the proposed kernel and it does not prevent the crash, which still occurs under the same circumstances when cryptsetup open runs. It's possible, however, that the patch decreases the likehood of a crash. In a very un-scientific test the latest patched kernel crashed in 3 out of 10 attempts, while the stock unpatched kernel crashed in 6 out of 10 attempts. To mitigate the issue, with both the original and patched kernels, I'm manually loading the dm-crypt module before running cryptsetup luksFormat. Note that manually loading the module between cryptsetup luksFormat and cryptsetup open does not prevent the crash. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
I've found other upstream fixes that might be interesting to test: https://git.launchpad.net/~arighi/+git/bionic-linux/log/?h=lp-1835279 I've uploaded a new test kernel here (still based of 4.15.0-54 + the backported fixes): https://kernel.ubuntu.com/~arighi/LP-1835279/ It'd be great if you could repeat your test and see if the NULL pointer dereference is still happening. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
The udev workaround preloading is unrelated and can be removed, crashes will also happen without using it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Attachment added: "4.15.0-54-generic #58+lp1835279 kernel log" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+attachment/5277230/+files/kernel-arighi.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
We're now opening the LUKS device using the master key directly and the crash is happening again in both the original 4.15.0-54-generic (buildd@lgw01-amd64-014) kernel or the patched 4.15.0-54-generic (arighi@kathleen) kernel. The backtrace is very similar but unlike the previous scenario it happens even if delays as large as 5000 ms are placed between partition table re-reading and commands. The LUKS device is created and accessed using the following commands: # cryptsetup -q luksFormat --type luks2 --pbkdf-memory 1 --master- key-file # LD_PRELOAD=/lib/no-udev.so cryptsetup open --master-key-file -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
Thanks Andrea. The new kernel seems to address the problem in such a way that the block device made available is still not immediately usable, but the kernel crash is prevented (at least in the test runs so far). Now errors can be handled from userspace to retry the operation in case of failure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
It would be interesting to try this fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37f9579f4c31a6d698dbf3016d7bf132f9288d30 I've uploaded a test kernel (based on 4.15.0-54 + the fix mentioned above): https://kernel.ubuntu.com/~arighi/LP-1835279/ Claudio, it would be great if you could do a test with this kernel. Thanks in advance! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Connor Kuehl (connork) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Connor Kuehl (connork) ** Changed in: linux (Ubuntu) Status: Confirmed => Invalid ** Changed in: linux (Ubuntu Bionic) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
Further testing suggests that the problem is caused by the kernel not having enough time to settle after partx -u exits (race between acknowledging the new block device and actually having it available?). Adding a 1s pause after the partition table is re-read seems to prevent this crash. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
This problem happens on Ubuntu Core (Core 20 development branch based on Core 18) and apport is not available. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1835279] Re: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open
** Description changed: This is Linux version 4.15.0-54-generic (buildd@lgw01-amd64-014) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 (Ubuntu 4.15.0-54.58-generic 4.15.18), from pc- kernel_240.snap Version signature: 4.15.0-54.58-generic 4.15.18 + Issue is non-deterministic, and happens in roughly 20% of the attempts. + Running on: qemu-kvm, command line: kvm \ - -bios /usr/share/ovmf/OVMF.fd \ - -smp 2 -m 512 -netdev user,id=mynet0,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 \ - -device virtio-net-pci,netdev=mynet0 \ - -drive file=pc.img,format=raw + -bios /usr/share/ovmf/OVMF.fd \ + -smp 2 -m 512 -netdev user,id=mynet0,hostfwd=tcp::8022-:22,hostfwd=tcp::8090-:80 \ + -device virtio-net-pci,netdev=mynet0 \ + -drive file=pc.img,format=raw Commands that caused the problem: cryptsetup -q --type luks2 --key-file luksFormat /dev/sda4 LD_PRELOAD=/lib/no-udev.so cryptsetup --type luks2 --key-file open /dev/sda4 crypt-data Notes: - See https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1589083 for more information on the no-udev workaround. - The commands are scripted. Also tried to add a 200ms and 1s interval before opening the device. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1835279 Title: 4.15.0-54.58-generic 4.15.18: oops/BUG on LUKS open To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835279/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
