[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-08-05 Thread Andrea Righi
** Description changed: - $ cat /proc/version_signature + [Impact] + + bcache_allocator() can call the following: + + bch_allocator_thread() + -> bch_prio_write() + -> bch_bucket_alloc() + -> wait on >set->bucket_wait + + But the wake up event on bucket_wait is

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-07-24 Thread Brad Figg
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to:

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-07-10 Thread Andrea Righi
** This bug is no longer a duplicate of bug 1796292 Tight timeout for bcache removal causes spurious failures ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Andrea Righi (arighi) ** Changed in: linux (Ubuntu Cosmic) Assignee: (unassigned) => Andrea Righi (arighi) **

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-07-05 Thread Andrea Righi
*** This bug is a duplicate of bug 1796292 *** https://bugs.launchpad.net/bugs/1796292 ** This bug has been marked a duplicate of bug 1796292 Tight timeout for bcache removal causes spurious failures -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-07-05 Thread Andrea Righi
Possibly a duplicate of bug 1796292, since we've got the same hung task timeout trace in bch_bucket_alloc() with the latest test kernel (4.15.0-55-generic #60+lp796292). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2019-01-22 Thread Joseph Salisbury via ubuntu-bugs
** Changed in: linux (Ubuntu) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Bionic) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Cosmic) Status: In Progress => Confirmed ** Changed in: linux (Ubuntu Cosmic) Assignee: Joseph Salisbury

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-27 Thread Ryan Harper
v4.13 shows the same search_free stack as 4.14, but doesn't appear to be fatal (it doesn't result in a hung process nor a block device that cannot be wiped. I suspect we've got a new upstream issue since 4.20 didn't fix things. Re-running 4.20 to see if we had a stack trace, we don't NULL pointer

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-27 Thread Ryan Harper
I also tested v4.15-rc5, which failed with the same stack trace. Then I returned to 4.14 mainline, and got this one: [ 953.659489] BUG: unable to handle kernel NULL pointer dereference at 0d40 [ 953.668707] IP:

Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-27 Thread Ryan Harper
It took a while, but v4.15-rc6 fails; so we need to move back a bit. [ 2541.139904] bcache: bcache_device_free() bcache0 stopped [ 2541.141848] BUG: unable to handle kernel NULL pointer dereference at 0d38 [ 2541.143813] IP: search_free+0ex01-b6435-e1/5f0x40 [bcache] [ 2541.145676]

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-26 Thread Joseph Salisbury
Can you confirm that v4.15-rc6 does not exhibit the bug? That will ensure I'm reviewing the correct commits: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc6/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-26 Thread Ryan Harper
The 4.15.0-39 kernel FAILs: + make-bcache -C /dev/sdb Can't open dev /dev/sdb: Device or resource busy + exit 1 Successful runs: 24 root@ubuntu:~# [ 242.655013] INFO: task kworker/3:103:2610 blocked for more than 120 seconds. [ 242.665486] Not tainted 4.15.0-39-generic #43~lp1784665 [

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-26 Thread Joseph Salisbury
I built a new Bionic test kernel. The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test this kernel and see if it resolves this bug? Note about installing test kernels: • If the test kernel is prior to 4.15(Bionic) you need to install the

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-26 Thread Ryan Harper
4.20-rc3 also fails. Successful runs: 95 root@ubuntu:~# [ 484.565298] INFO: task kworker/3:206:2091 blocked for more than 120 seconds. [ 484.566976] Not tainted 4.20.0-042000rc3-generic #201811182231 [ 484.568069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-19 Thread Joseph Salisbury
The 4.20 kernel is now available. Can you give v4.20-rc3 a test: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-11-14 Thread Joseph Salisbury
** Tags removed: kernel-key ** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-29 Thread Joseph Salisbury
Thanks for testing. I was hoping a fix would be in 4.19, but I'll dig some more. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage

Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-29 Thread Ryan Harper
On Wed, Aug 29, 2018 at 12:01 PM Joseph Salisbury wrote: > > Thanks for digging up this additional information. I'll investigate > further. While I do that, the v4.19-rc1 kernel is now out. It might be > worthwhile to give that one a go, and see if the fix was already > committed to mainline:

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-29 Thread Joseph Salisbury
Thanks for digging up this additional information. I'll investigate further. While I do that, the v4.19-rc1 kernel is now out. It might be worthwhile to give that one a go, and see if the fix was already committed to mainline: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc1/ -- You

Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Ryan Harper
https://www.spinics.net/lists/linux-bcache/msg04869.html https://www.spinics.net/lists/linux-bcache/msg05774.html On Tue, Aug 28, 2018 at 2:30 PM Ryan Harper wrote: > > It's always been a locking issue, so either misuse (or missing) locks > in the bcache attach/detach path, or generic locking

Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Ryan Harper
It's always been a locking issue, so either misuse (or missing) locks in the bcache attach/detach path, or generic locking changes to block layer path I'd think. I'll see if I can find any hits on those oops tracebacks too On Tue, Aug 28, 2018 at 2:29 PM Ryan Harper wrote: > >

Re: [Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Ryan Harper
https://patchwork.kernel.org/patch/10094201/ diff --git a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts b/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts index d4f80786e7c2..3890468678ce 100644 --- a/arch/arm64/boot/dts/rockchip/rk3328-rock64.dts +++

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Joseph Salisbury
The bisect reported the following commit as the first bad commit: bc631943faba ("arm64: dts: rockchip: limit rk3328-rock64 gmac speed to 100MBit for now") It's strange that this is an arm64 commit. I'll investigate a bit and see why the bisect came back with this result. -- You received this

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Ryan Harper
FAILED: Kernel 4.15-rc1 up to bc631943faba6fc3f755748091ada31798fb7d50 Attached is the oops of the failure. ** Attachment added: "rc1-bisect7-oops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5181823/+files/rc1-bisect7-oops.txt -- You received this bug

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: bc631943faba6fc3f755748091ada31798fb7d50 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Ryan Harper
PASS: Kernel 4.15-rc1 (bisect6) up to 87eba0716011e528f7841026f2cc65683219d0ad -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-28 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: 87eba0716011e528f7841026f2cc65683219d0ad The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-27 Thread Ryan Harper
PASS: Kernel 4.15-rc1 (bisect5) up to 6b457409169b7686d293b408da0b6446ccb57a76 I've 2600 seconds of uptime with over 800 loops in the test-case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-27 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: 6b457409169b7686d293b408da0b6446ccb57a76 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-27 Thread Ryan Harper
FAILED: Kernel 4.15-rc1 up to f48f66a962a54c3d26d688c3df5441c9d1ba8730 Attached is the oops of the failure. ** Attachment added: "rc1-bisect4-oops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5181417/+files/rc1-bisect4-oops.txt -- You received this bug

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-22 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: f48f66a962a54c3d26d688c3df5441c9d1ba8730 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-21 Thread Ryan Harper
FAILED: Kernel 4.15-rc1 up to f17b9e764dfcd838dab51572d620a371c05a8e60 Attached is the oops of the failure. ** Attachment added: "rc1-bisect3-ops.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5178807/+files/rc1-bisect3-ops.txt -- You received this bug

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-21 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: f17b9e764dfcd838dab51572d620a371c05a8e60 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-15 Thread Ryan Harper
test kernel 4.15-rc1 up to abb62c46d4949d44979fa647740feff3f7538799 FAILED ** Attachment added: "kernel oops for 4.15-rc1 up to abb62c46d4949d44979fa647740feff3f7538799" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5175947/+files/rc1-bisect2-oops.txt -- You

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-14 Thread Joseph Salisbury
I built the next test kernel, up to the following commit: abb62c46d4949d44979fa647740feff3f7538799 The test kernel can be downloaded from: http://kernel.ubuntu.com/~jsalisbury/lp1784665 Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-14 Thread Ryan Harper
test kernel 4.15-rc6 up to d8887f1c7289848e74c92bd4322789a9cd7de699 FAILED ** Attachment added: "kernel oops for 4.15-rc6-up-to-d8887f1c7289848e74c92bd4322789a9cd7de699" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5175379/+files/rc6-bisect1-oops.txt -- You

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-14 Thread Ryan Harper
test kernel 4.15-rc6 up to d8887f1c7289848e74c92bd4322789a9cd7de699 FAILED Attaching oops. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-14 Thread Joseph Salisbury
I started a kernel bisect between v4.15-rc6 and v4.15-rc7. The kernel bisect will require testing of about 7-10 test kernels. I built the first test kernel, up to the following commit: d8887f1c7289848e74c92bd4322789a9cd7de699 The test kernel can be downloaded from:

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-14 Thread Joseph Salisbury
** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Status: New => In Progress ** Changed in: linux (Ubuntu Cosmic) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu Bionic) Importance: Undecided =>

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-13 Thread Ryan Harper
4.14 Final: OK 4.15-rc1:OK I started a further bisect: 4.15-rc9:FAIL 4.15-rc5:OK 4.15-rc7:FAIL 4.15-rc6:OK So, it appears something between rc6 (last known good) and rc7 (FAIL) -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-13 Thread Joseph Salisbury
To narrow it down further, can you test the following two kernels: 4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/ 4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-13 Thread Ryan Harper
4.10: Status: OK Comment: Survived 30 minute loop over recreating with no issues. 4.13: Status: OK Comment: Survived 30 minute loop over recreating with no issues. 4.15: Status: FAIL Comment: triggered within less than a minute of the loop. Kernel Trace: [ 122.500943] Oops:

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-09 Thread Joseph Salisbury
Thanks for testing mainline. We should be able to bisect this down if we confirm it's a regression. Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug: 4.10 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10/ 4.13

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Ryan Harper
I can recreate on upstream: # uname -r; dmesg | tail -n 16 4.18.0-041800rc7-generic [ 967.657492] INFO: task bcache_allocato:14240 blocked for more than 120 seconds. [ 967.664138] Not tainted 4.18.0-041800rc7-generic #201807292230 [ 967.669641] "echo 0 >

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Ryan Harper
I'll see if I can reproduce on the mainline kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To manage notifications about this bug go to:

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Ryan Harper
I think I've got one. It doesn't always hang the mkfs, but it does wedge bcache in a few ways. [ 484.322546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.326230] INFO: task bcache_allocato:32503 blocked for more than 120 seconds. 1) when it occurs, a

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Ryan Harper
** Attachment added: "bcache-basic-repro.sh" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784665/+attachment/5170528/+files/bcache-basic-repro.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Joseph Salisbury
It would be great if we can identify a reproducer. It would make it much easier to track down. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1784665 Title: mkfs.ext4 over /dev/bcache0 hangs To

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-08-01 Thread Ryan Harper
We've just had a second occurrence of this, same kernel, but with Xenial. (Xenial HWE Edge kernel 4.15.0-29) with the same scenario, mkfs.ext4 a /dev/bcache0. I'll see if I can recreate by stressing things. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-07-31 Thread Ryan Harper
> Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? We've had plenty of successful runs on this kernel. I suspect io load or something else may trigger this. We've only seen it once out of tens of not

[Bug 1784665] Re: mkfs.ext4 over /dev/bcache0 hangs

2018-07-31 Thread Joseph Salisbury
Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem? Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.18