On Wed, Mar 06, 2019 at 07:55:45AM -0600, Russell Mosemann wrote:
>The four most problematic servers were hanging nearly every day. They each
>have a dedicated hard drive and are running 4.19.0-0.bpo.2-amd64. I
>installed btrfs-progs 4.20.1 on each and recreated the file system on the
The four most problematic servers were hanging nearly every day. They each have
a dedicated hard drive and are running 4.19.0-0.bpo.2-amd64. I installed
btrfs-progs 4.20.1 on each and recreated the file system on the hard drive.
There have been no issues since then, but it is too early to
On Sat, Mar 02, 2019 at 05:12:42PM -0600, Russell Mosemann wrote:
>I used btrfs-progs (4.17-1~bpo9+1) from the backports repository, which
>does have zstd support. btrfs-check does need zstd support. Otherwise, it
>will error out with unsupported feature (10).
>
Unfortunately
y, March 2, 2019 3:59pm
To: "Russell Mosemann" , 908...@bugs.debian.org
Subject: Re: Bug#908216: btrfs blocked for more than 120 seconds
Hi Russell,
The backport of btrfs-progs is old and doesn't have libzstd support,
and I'm still waiting for a sponsor (#922951), but if you'd like to
Hi Russell,
The backport of btrfs-progs is old and doesn't have libzstd support,
and I'm still waiting for a sponsor (#922951), but if you'd like to
give it a try before it hits the archive:
dget
https://mentors.debian.net/debian/pool/main/b/btrfs-progs/btrfs-progs_4.20.1-2~bpo9+1.dsc
cd
We continue to see hangs on one or more servers every night. It happens when
copying a file into the btrfs partition. The result of the hang is that the CPU
goes to 50% I/O wait or higher for hours. Only a reboot resets the process.
Based on this behavior and the kernel errors, it looks like
I ran a "btrfs check" on one of the partitions that had issues last night to
confirm that the file system was not damaged. No errors were found.
# btrfs check -p /dev/sdc1
Checking filesystem on /dev/sdc1
UUID: d6640b4b-bd59-47d5-bfd7-37b4c40fb3c6
checking extents [.]
checking free space cache
Two servers hung last night. Both events happened while a file was being copied
into the btrfs file system. No references or vm's were involved. It was a
simple file copy.
I had the opportunity to observe the system while it happened. It turns out
that the CPU time goes to 50% or more I/O
On Monday, February 25, 2019 10:17pm, "Nicholas D Steeves"
said:
> Control: tags -1 -unreproducible
>
> Hi Russell,
>
> Thank you for providing more info. Now I see where you're running
> into known limitations with btrfs (all versions). Reply follows inline.
>
> BTW, you're not using SMR
Control: tags -1 -unreproducible
Hi Russell,
Thank you for providing more info. Now I see where you're running
into known limitations with btrfs (all versions). Reply follows inline.
BTW, you're not using SMR and/or USB disks, right?
On Mon, Feb 25, 2019 at 12:33:51PM -0600, Russell Mosemann
Steps to reproduce
Simply copying a file into the file system can cause things to lock up. In this
case, the files will usually be thin-provisioned qcow2 disks for kvm vm's.
There is no detailed formula to force the lockup to occur, but it happens
regularly, sometimes multiple times in one
Control: tags -1 + unreproducible moreinfo
Hi Russell,
On Sun, Feb 24, 2019 at 06:34:59PM -0600, Russell Mosemann wrote:
>
>
>Package: src:linux
>Version: 4.19.16-1~bpo9+1
>Severity: important
>
>
>
>
>
>If the btrfs fixes will not be backported to 4.19, then
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
If the btrfs fixes will not be backported to 4.19, then this is not an
appropriate base kernel for the next version of Debian.
Feb 24 19:09:21 vhost182 kernel: INFO: task btrfs-transacti:604 blocked for
more than 120
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 24 17:19:32 vhost002 kernel: INFO: task btrfs-transacti:643 blocked for
more than 120 seconds.
Feb 24 17:19:32 vhost002 kernel: Not tainted 4.19.0-0.bpo.2-amd64 #1
Debian 4.19.16-1~bpo9+1
Feb 24 17:19:32 vhost002
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 24 10:47:24 lxc008 kernel: INFO: task btrfs:4752 blocked for more than 120
seconds.
Feb 24 10:47:25 lxc008 kernel: Tainted: G I
4.19.0-0.bpo.2-amd64 #1 Debian 4.19.16-1~bpo9+1
Feb 24 10:47:25 lxc008
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 23 00:36:42 vhost002 kernel: INFO: task btrfs-transacti:638 blocked for
more than 120 seconds.
Feb 23 00:36:42 vhost002 kernel: Not tainted 4.19.0-0.bpo.2-amd64 #1
Debian 4.19.16-1~bpo9+1
Feb 23 00:36:42 vhost002
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 22 19:09:23 vhost182 kernel: INFO: task btrfs-transacti:612 blocked for
more than 120 seconds.
Feb 22 19:09:23 vhost182 kernel: Not tainted 4.19.0-0.bpo.2-amd64 #1
Debian 4.19.16-1~bpo9+1
Feb 22 19:09:23 vhost182
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
This is a frequent bug that occurs nearly daily for many months, now.
Feb 21 19:02:38 vhost182 kernel: INFO: task btrfs-transacti:617 blocked for
more than 120 seconds.
Feb 21 19:02:38 vhost182 kernel: Not tainted
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 21 17:19:15 vhost002 kernel: INFO: task btrfs-transacti:636 blocked for
more than 120 seconds.
Feb 21 17:19:15 vhost002 kernel: Not tainted 4.19.0-0.bpo.2-amd64 #1
Debian 4.19.16-1~bpo9+1
Feb 21 17:19:15 vhost002
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 20 18:34:42 lxc008 kernel: INFO: task fbackup main:27837 blocked for more
than 120 seconds.
Feb 20 18:34:58 lxc008 kernel: Tainted: G I
4.19.0-0.bpo.2-amd64 #1 Debian 4.19.16-1~bpo9+1
Feb 20
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
Feb 19 19:01:04 vhost182 kernel: INFO: task btrfs-transacti:613 blocked for
more than 120 seconds.
Feb 19 19:01:04 vhost182 kernel: Tainted: GE
4.19.0-0.bpo.1-amd64 #1 Debian 4.19.12-1~bpo9+1
Feb 19
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 15 14:21:09 lxc009 kernel: INFO: task rsync:1762 blocked for more than 120
seconds.
Feb 15 14:21:09 lxc009 kernel: Not tainted 4.19.0-0.bpo.2-amd64 #1 Debian
4.19.16-1~bpo9+1
Feb 15 14:21:09 lxc009 kernel: "echo 0 >
Package: src:linux
Version: 4.19.16-1~bpo9+1
Severity: important
Feb 15 08:08:50 lxc008 kernel: INFO: task fbackup main:2330 blocked for more
than 120 seconds.
Feb 15 08:08:50 lxc008 kernel: Tainted: G I
4.19.0-0.bpo.2-amd64 #1 Debian 4.19.16-1~bpo9+1
Feb 15
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
With the debilitating btrfs bugs in 4.19, it probably wasn't prudent to remove
4.17 and 4.18 from the repository, until 4.20 is released.
Jan 25 17:17:50 vhost002 kernel: INFO: task btrfs-transacti:661 blocked for
more than
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
Jan 23 17:16:23 vhost002 kernel: INFO: task btrfs-transacti:638 blocked for
more than 120 seconds.
Jan 23 17:16:23 vhost002 kernel: Tainted: GE
4.19.0-0.bpo.1-amd64 #1 Debian 4.19.12-1~bpo9+1
Jan 23
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
Jan 22 09:35:31 vhost002 kernel: INFO: task btrfs-transacti:4031 blocked for
more than 120 seconds.
Jan 22 09:35:31 vhost002 kernel: Tainted: GE
4.19.0-0.bpo.1-amd64 #1 Debian 4.19.12-1~bpo9+1
Jan 22
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
The only solution is to revert to 4.17.
Jan 19 18:14:35 vhost004 kernel: INFO: task btrfs-transacti:691 blocked for
more than 120 seconds.
Jan 19 18:14:35 vhost004 kernel: Tainted: G I E
4.19.0-0.bpo.1-amd64
Package: src:linux
Version: 4.19.12-1~bpo9+1
Severity: important
Yes, the btrfs hung task problem still exists in the latest kernel.
Jan 19 18:40:41 vhost003 kernel: INFO: task btrfs-transacti:666 blocked for
more than 120 seconds.
Jan 19 18:40:41 vhost003 kernel: Tainted: G I E
Package: src:linux
Version: 4.18.20-2~bpo9+1
Severity: important
This problem was so bad with 4.18.6-1~bpo9+1 that I had to revert to 4.17.0.
When saw that 4.18.20 was out, I wanted to give it a try. Big mistake. Please
fix this problem, already.
Dec 22 18:14:11 vhost004 kernel: INFO: task
Package: src:linux
Version: 4.18.6-1~bpo9+1
Severity: important
Nov 01 03:43:58 lxc009 kernel: INFO: task rsync:39988 blocked for more than 120
seconds.
Nov 01 03:43:58 lxc009 kernel: Not tainted 4.18.0-0.bpo.1-amd64 #1 Debian
4.18.6-1~bpo9+1
Nov 01 03:43:58 lxc009 kernel: "echo 0 >
Package: src:linux
Version: 4.18.6-1~bpo9+1
Severity: important
Oct 19 02:38:41 lxc009 kernel: INFO: task rsync:18730 blocked for more than 120
seconds.
Oct 19 02:38:41 lxc009 kernel: Not tainted 4.18.0-0.bpo.1-amd64 #1 Debian
4.18.6-1~bpo9+1
Oct 19 02:38:41 lxc009 kernel: "echo 0
Package: src:linux
Version: 4.18.6-1~bpo9+1
Severity: important
Oct 18 01:40:34 lxc009 kernel: INFO: task kworker/u130:2:40876 blocked for more
than 120 seconds.
Oct 18 01:40:34 lxc009 kernel: Not tainted 4.18.0-0.bpo.1-amd64 #1 Debian
4.18.6-1~bpo9+1
Oct 18 01:40:34 lxc009
Package: src:linux
Version: 4.18.6-1~bpo9+1
Severity: important
Oct 17 01:07:25 lxc008 kernel: INFO: task fbackup main:23119 blocked for more
than 120 seconds.
Oct 17 01:07:25 lxc008 kernel: Tainted: G I
4.18.0-0.bpo.1-amd64 #1 Debian 4.18.6-1~bpo9+1
Oct 17
Package: src:linux
Version: 4.18.6-1~bpo9+1
Severity: important
This problem is happening on a daily basis on more than one machine.
Oct 16 01:12:41 lxc008 kernel: INFO: task rsync:14731 blocked for more than 120
seconds.
Oct 16 01:12:41 lxc008 kernel: Tainted: G I
Package: linux-image-4.18.0-0.bpo.1-amd64
Severity: important
Sep 27 01:25:47 lxc008 kernel: INFO: task rsync:24406 blocked for more than 120
seconds.
Sep 27 01:25:47 lxc008 kernel: Tainted: G I
4.18.0-0.bpo.1-amd64 #1 Debian 4.18.6-1~bpo9+1
Sep 27 01:25:47 lxc008
Package: linux-image-4.18.0-0.bpo.1-amd64
Severity: important
Sep 25 22:10:29 lxc008 kernel: INFO: task fbackup main:6251 blocked for more
than 120 seconds.
Sep 25 22:10:30 lxc008 kernel: Tainted: G I
4.18.0-0.bpo.1-amd64 #1 Debian 4.18.6-1~bpo9+1
Sep 25 22:10:30 lxc008
Package: linux-image-4.18.0-0.bpo.1-amd64
Severity: important
Sep 22 17:15:44 lxc008 kernel: INFO: task backup cleanup:2165 blocked for more t
han 120 seconds.
Sep 22 17:15:44 lxc008 kernel: Tainted: G I 4.18.0-0.bpo.1-
amd64 #1 Debian 4.18.6-1~bpo9+1
Sep 22 17:15:44 lxc008
Package: linux-image-4.17.0-0.bpo.3-amd6
Severity: important
Sep 09 00:39:52 lxc008 kernel: INFO: task rsync:20113 blocked for more than 120
seconds.
Sep 09 00:39:52 lxc008 kernel: Tainted: G I
4.17.0-0.bpo.3-amd64 #1 Debian 4.17.17-1~bpo9+1
Sep 09 00:39:52 lxc008
38 matches
Mail list logo
