[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
** Tags added: cscc
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
** Changed in: linux (Ubuntu Trusty)
Assignee: Joseph Salisbury (jsalisbury) => (unassigned)
** Changed in: linux (Ubuntu)
Assignee: Joseph Salisbury (jsalisbury) => (unassigned)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
Joseph,
I'm currently testing a 4.15.0-13 kernel from xenial-16.04-edge path on
these hosts. I just had the issue exhibit before the kernel change, so
we should know within a couple days if that helps. Unfortunately, the
logs for this system beyond those shared are not available publicly.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
@Drew
Do you have an update to the request above?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
Commit 9baf30972b55 is in the Xenial kernel as of 4.4.0-98. It sounds
like you are running the 4.4.0-111 kernel, correct?
The second patch you mention was never accepted into mainline. We can
test it out. However, it might be good to first test the mainline
kernel to see if some other commit already fixed this bug. Can you test
the current mainline kernel? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc6
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
** Changed in: linux (Ubuntu)
Assignee: (unassigned) => Joseph Salisbury (jsalisbury)
** Changed in: linux (Ubuntu Trusty)
Assignee: (unassigned) => Joseph Salisbury (jsalisbury)
** Changed in: linux (Ubuntu)
Status: Triaged => In Progress
** Changed in: linux (Ubuntu Trusty)
Status: Triaged => In Progress
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1757277] Re: soft lockup from bcache leading to high load and lockup on trusty
** Tags added: kernel-da-key
** Changed in: linux (Ubuntu)
Status: Incomplete => Triaged
** Changed in: linux (Ubuntu)
Importance: Undecided => High
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Trusty)
Status: New => Triaged
** Changed in: linux (Ubuntu Trusty)
Importance: Undecided => High
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1757277
Title:
soft lockup from bcache leading to high load and lockup on trusty
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Trusty:
In Progress
Bug description:
I have an environment with Dell R630 servers with RAID controllers
with two virtual disks and 22 passthru devices. 2 SAS SSDs and 20
HDDs are setup in 2 bcache cachesets with a resulting 20 mounted xfs
filesystems running bcache backending an 11 node swift cluster (one
zone has 1 fewer nodes). Two of the zones have these nodes as
described above and they appear to be exibiting soft lockups in the
bcache thread of the kernel causing other kernel threads to go into
i/o blocking state an keeping processes on any bcache from being
successful. disk access to the virtual disks mounted with out bcache
is still possible when this lockup occurs.
https://pastebin.ubuntu.com/p/mtn47QqBJ3/
There are several softlockup messages found in the dmesg and many of
the dumpstack are locked inside the bch_writeback_thread();
static int bch_writeback_thread(void *arg)
{
[...]
while (!kthread_should_stop()) {
down_write(>writeback_lock);
[...]
}
One coredump is found when the kswapd is doing the reclaim about the
xfs inode cache.
__xfs_iflock(
struct xfs_inode *ip)
{
do {
prepare_to_wait_exclusive(wq, , TASK_UNINTERRUPTIBLE);
if (xfs_isiflocked(ip))
io_schedule();
} while (!xfs_iflock_nowait(ip));
- Possible fix commits:
1). 9baf30972b55 bcache: fix for gc and write-back race
https://www.spinics.net/lists/linux-bcache/msg04713.html
- Related discussions:
1). Re: [PATCH] md/bcache: Fix a deadlock while calculating writeback rate
https://www.spinics.net/lists/linux-bcache/msg04617.html
2). Re: hang during suspend to RAM when bcache cache device is attached
https://www.spinics.net/lists/linux-bcache/msg04636.html
We are running trusty/mitaka swift storage on these nodes with
4.4.0-111 kernel (linux-image-generic-lts-xenial).
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1757277/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
