[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Updated bug description and added Bionic bug task. Marked bug 1785081
as a duplicate of this bug.
** Also affects: linux (Ubuntu Bionic)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Bionic)
Status: New => Triaged
** Changed in: linux (Ubuntu Bionic)
Importance: Undecided => Critical
** Tags added: kernel-da-key
** Summary changed:
- Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for
a very long time.
+ Ubuntu 18.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for
a very long time.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 18.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Status in linux source package in Zesty:
Won't Fix
Status in linux source package in Bionic:
Triaged
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2.
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
But that new lp bug is sparse on details too can all comments be
pushed through on
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1785081 ?
What exactly is happening with the bionic kernel, and what is the
expectation here?
The comments in this bug report from cking still stand imho. Unless
there is something else new?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Dimitri, we've synced the other bug 1785081 that is probably the same
issue as this one.
F.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
This bug was closed in january 2018. Are you sure this is the same issue
/ right ticket? I see that you have recently marked some other bug as a
duplicate of this one. maybe you can sync that one? If not please
state what is happening on 18.04 since in january it was believed that
this issue was fixed.
** Changed in: linux (Ubuntu)
Status: Invalid => New
** Changed in: linux (Ubuntu)
Importance: Critical => Undecided
** Changed in: linux (Ubuntu)
Assignee: Colin Ian King (colin-king) => (unassigned)
** Changed in: linux (Ubuntu Zesty)
Assignee: Colin Ian King (colin-king) => (unassigned)
** Changed in: ubuntu-power-systems
Status: Invalid => New
** Changed in: ubuntu-power-systems
Importance: Critical => Undecided
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX,
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
For now reopening the states of this issue in launchpad as NEW and
UNDECIDED priority.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some time)
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Tags removed: severity-critical
** Tags added: severity-high
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
Invalid
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some time)
B. htx
C.
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
--- Comment From dougm...@us.ibm.com 2018-05-14 13:40 EDT---
Note, we are still seeing issues with CFQ on 18.04 (Bionic).
** Tags removed: targetmilestone-inin1704
** Tags added: targetmilestone-inin1804
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
Invalid
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
These patches are available in Artful and will be available in the
linux-hwe kernel, please upgrade to the latest linux-hwe and reopen this
bug if you are able to reproduce it. Marking it invalid for now.
** Changed in: linux (Ubuntu)
Status: In Progress => Invalid
** Changed in: ubuntu-power-systems
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
Invalid
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Ubuntu 17.04 has reached end of life. No further bugfixes will be
applied to this version.
** Changed in: linux (Ubuntu Zesty)
Status: In Progress => Won't Fix
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
Incomplete
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
Won't Fix
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Marking as "incomplete" until fix lands upstream.
** Changed in: ubuntu-power-systems
Status: In Progress => Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
Incomplete
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Tags removed: triage-r
** Tags added: triage-g
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some time)
B. htx
C. Select
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Tags added: triage-r
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some time)
B. htx
C. Select the test file "mdt.io"
D.
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Thanks for also testing with the latest upstream kernel and reporting it
to the upstream developers. I've added the link to the bug so we pull in
bug updates automatically from the bugzilla.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
Doug,
"Can we get some answers as to why CFQ is the default scheduler? It
seems like the expedient fix is to change the default to "deadline"."
Sure. We went back to CFQ as it was showing to be a good general purpose
I/O scheduler for a lot of wide ranging and general purpose I/O demands.
Unfortunately you seem to have uncovered some cases where clearly it
does not behave well for your use case where deadline would be more
appropriate. Like all schedulers, it's hard to chose one that is going
to meet every use case and every scenario perfectly, especially for a
wide range of devices, device configurations, system memory size and
file system choices.
We do try to systematically check the performance and latencies of
various synthetic test scenarios across a range of file systems on each
kernel to try and spot issues: http://kernel.ubuntu.com/~cking/fs-
tests/jun-2017/
CFQ has several tunables that may help your issue, I'd refer you to:
https://www.kernel.org/doc/Documentation/block/cfq-iosched.txt
it may be worth reviewing this document and then seeing if the CFQ can
be tuned to improve the issues you are seeing.
** Also affects: linux via
https://bugzilla.kernel.org/show_bug.cgi?id=196737
Importance: Unknown
Status: Unknown
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in Linux:
Unknown
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
--- Comment From dougm...@us.ibm.com 2017-08-23 09:19 EDT---
Opened kernel.org bugzilla https://bugzilla.kernel.org/show_bug.cgi?id=196737
** Bug watch added: Linux Kernel Bug Tracker #196737
https://bugzilla.kernel.org/show_bug.cgi?id=196737
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
I've rolled three patches into a test kernel to test. I believe the best
fix is probably "cfq: Disable writeback throttling by default"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=142bbdfccc8b3e9f7342f2ce8422e76a3b45beae
The commits are:
4d608baac5f4e72b033a122b2d6d9499532c3afc "block: Initialize cfqq->ioprio_class
in cfq_get_queue()"
142bbdfccc8b3e9f7342f2ce8422e76a3b45beae "cfq: Disable writeback throttling by
default"
5be6b75610cefd1e21b98a218211922c2feb6e08 "cfq-iosched: fix the delay of
cfq_group's vdisktime under
iops mode"
You can download the packages from:
http://kernel.ubuntu.com/~cking/lp1709889/
Let me know if this helps with the issue.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Changed in: linux (Ubuntu)
Assignee: Joseph Salisbury (jsalisbury) => Colin Ian King (colin-king)
** Changed in: linux (Ubuntu Zesty)
Assignee: Joseph Salisbury (jsalisbury) => Colin Ian King (colin-king)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
I built a 17.04(Zesty) test kernel with a pick of commit 5be6b75610ce.
This kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1709889/
Can you test this kernel and see if it resolves this bug?
I also see that commit 5be6b75610ce has been cc'd to upstream stable,
but it has not landed in upstream 4.10.y yet.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Changed in: ubuntu-power-systems
Status: New => In Progress
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
In Progress
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
leaving the console free to watch for messages.
To run:
A. su - htx (this may take some time)
B. htx
C. Select the test
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Changed in: linux (Ubuntu)
Status: New => In Progress
** Changed in: linux (Ubuntu)
Importance: Undecided => Critical
** Changed in: linux (Ubuntu)
Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) =>
Joseph Salisbury (jsalisbury)
** Also affects: linux (Ubuntu Zesty)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Zesty)
Status: New => In Progress
** Changed in: linux (Ubuntu Zesty)
Importance: Undecided => Critical
** Changed in: linux (Ubuntu Zesty)
Assignee: (unassigned) => Joseph Salisbury (jsalisbury)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Zesty:
In Progress
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
[Kernel-packages] [Bug 1709889] Re: Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to adapter for a very long time.
** Also affects: ubuntu-power-systems
Importance: Undecided
Status: New
** Changed in: ubuntu-power-systems
Importance: Undecided => Critical
** Changed in: ubuntu-power-systems
Assignee: (unassigned) => Canonical Kernel Team (canonical-kernel-team)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1709889
Title:
Ubuntu 17.04: Bug in cfq scheduler, I/Os do not get submitted to
adapter for a very long time.
Status in The Ubuntu-power-systems project:
New
Status in linux package in Ubuntu:
New
Bug description:
---Problem Description---
When running stress test, sometimes seeing IO hung in dmesg or seeing "Host
adapter abort request" error.
---Steps to Reproduce---
There are two ways to re-create the issues:
(1)running HTX, you will see IO timeout backtrace in dmesg in several hours
(2)running some IO test, then reboot system, repeat this two steps, it takes
long time to re-create the issue.
---uname output---
4.10.0-11-generic
The bulk of the effort for this issue is currently being worked in
MicroSemi's JIRA https://jira.pmcs.com/browse/ESDIBMOP-133.
Ran an interesting test: Ran HTX until I started getting the "stall"
messages on the console, then shutdown HTX and examined the I/O
counters for the tested disks in sysfs:
root@bostonp15:~# for i in
/sys/devices/pci0003:00/0003:00:00.0/0003:01:00.0/host0/target0:2:[2345]/0:2:[2345]:0;
do echo ${i##*/} $(<${i}/iorequest_cnt) $(<${i}/iodone_cnt); done
0:2:2:0 0x5eba3d 0x5eba3d
0:2:3:0 0x773cc9 0x773cc9
0:2:4:0 0x782c61 0x782c61
0:2:5:0 0x5ca134 0x5ca134
root@bostonp15:~#
So, none of the disks showed any evidence of having lost an I/O. I
then restarted HTX and aside from having to manually restart one of
the disks, see no problems with the testing. It appears that what was
"hung" was purely in userland.
This does not absolve the kernel or aacraid driver from blame, but it
shows that the OS "believes" that it completed the I/O and thus
removed it from the queue. What we don't know is whether the OS truly
notified HTX about the completion, or if HTX (or userland libraries)
just failed to process the notification.
Tests are running again, will see what happens next.
Update from JIRA:
I have run some more experiments. Not sure what it tells us, but
here's what I've seen.
First test, ran until I got kernel messages about stalled tasks, then
shutdown HTX. After HTX was down, I checked the above mentioned
counters and found that on each disk iorequest_cnt matched iodone_cnt.
The disks were usable and I could restart HTX. This suggests that the
problem is not in the PM8069 firmware, and makes the case for the
aacraid driver having a bug somewhat weaker. However, this merely says
that the driver "completed" the I/O as far as the kernel is concerned,
not that a completion rippled back to the application.
I restarted HTX and have run until errors. This time, I am leaving HTX
running and observing. Two of the disks reached the HTX error
threshold and the testers stopped (those 2 disks are now idle).
Another disks saw errors but then stopped and appears to be running
fine now. The last disk has not seen any errors (yet). On the two idle
(errored-out) disks I see iorequest_cnt matches iodone_cnt. I am able
to "terminate and restart" the two idle disks and HTX appears to be
testing them again "normally". Note that no reboot was required,
further supporting the evidence that, as far as the kernel is
concerned, there is nothing wrong with the disks and their I/O paths.
So, I don't believe this completely eliminates aacraid from the
picture, especially given we don't see this behavior on other
systems/drivers. But, it probably moves the focus of the investigation
away form the adapter firmware.
Tried build upstream 4.11 kernel on Ubuntu. This still gets the hangs.
Both Ubuntu 4.10 and upstream 4.11 have aacraid driver
1.2.1[50792]-custom.
Good new/bad news... While doing an initial evaluation of the LSI-3008
SAS HBA on Boston and Ubuntu 17.04, I am hitting this same problem.
So, it appears to have nothing specific to do with the PM8069 or
aacraid driver.
Some notes on reproduce this. I have been using the github release of
HTX, built using the following steps:
1. apt install make gcc g++ git libncurses5-dev libcxl-dev libdapl-dev
(others may be required)
2. git clone https://github.com/open-power/HTX
3. cd HTX
4. make
5. make deb
Then install the resulting "htxubuntu.deb" package.
Note, HTX will not test disks that have a filesystem or OS installed,
so there must be at least two disks made available to HTX by clearing
any previous data. A partition table is optional, in my testing I have
none.
Also, it may be desirable to run HTX somewhere other than the console,
