Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

[Byungchul Park]

On Sat, Dec 16, 2017 at 6:15 AM, Theodore Ts'o  wrote:
> On Fri, Dec 15, 2017 at 05:39:25PM +0900, Byungchul Park wrote:
>>
>> All locks should belong to one class if each path of acquisition
>> can be switchable each other within the class at any time.
>> Otherwise, they should belong to a different class.
>
> OK, so let's go back to my case of a Network Block Device with a local
> file system mounted on it, which is then exported via NFS.
>
> So an incoming TCP packet can go into the NFS server subsystem, then
> be processed by local disk file system, which then does an I/O
> operation to the NBD device, which results in an TCP packet being sent
> out.  Then the response will come back over TCP, into the NBD block
> layer, then into the local disk file system, and that will result in
> an outgoing response to the TCP connection for the NFS protocol.
>
> In order to avoid cross release problems, all locks associated with
> the incoming TCP connection will need to be classified as belonging to
> a different class as the outgoing TCP connection.  Correct?  One
> solution might be to put every single TCP connection into a separate
> class --- but that will explode the number of lock classes which
> Lockdep will need to track, and there is a limited number of lock
> classes (set at compile time) that Lockdep can track.  So if that
> doesn't work, we will have to put something ugly which manually makes
> certain TCP connections "magic" and require them to be put into a
> separate class than all other TCP connections, which will get
> collapsed into a single class.  Basically, any TCP connection which is
> either originated by the kernel, or passed in from userspace into the
> kernel and used by some kernel subsystem, will have to be assigned its
> own lockdep class.
>
> If the TCP connection gets closed, we don't need to track that lockdep
> class any more.  (Or if a device mapper device is torn down, we
> similarly don't need any unique lockdep classes created for that
> device mapper device.)  Is there a way to tell lockdep that a set of
> lockdep classes can be released so we can recover the kernel memory to
> be used to track some new TCP connection or some new device mapper
> device?

Right. I also think lockdep should be able to reflect that
kind of dynamic situations to do a better job.

The fact that kernel works well w/o that work doesn't
mean current status is perfect, in my opinion.

As you know, lockdep is running within very limited
environment so it's very hard to achieve it.

However, anyway, I think that's a problem and should
be solved by modifying lockdep core. Actually, that had
been one on my to-dos, if allowed.

For some waiters, for which this is only solution to play
with cross-release, I think we can invalidate those
waiters for now, while all others still get benefit.

We have added acquire annotations manually to
consider waiters one by one, and I am sure it's going
to continue in the future.

IMO, considering all waiters at once and fixing false
positives in a right way or invalidating one by one is
better than considering waiters one by one as is, of
course, while keeping off by default.

-- 
Thanks,
Byungchul

Re: [GIT PULLL] nvme fixes for Linux 4.15

[Jens Axboe]

On 12/15/2017 09:48 AM, Christoph Hellwig wrote:
> Hi Jens,
> 
> below are a couple important one or two liners fixes for Linux 4.15:
> 
> 
> The following changes since commit fcf38cdf332a81b20a59e3ebaea81f6b316bbe0c:
> 
>   kyber: fix another domain token wait queue hang (2017-12-06 12:33:07 -0700)
> 
> are available in the git repository at:
> 
>   git://git.infradead.org/nvme.git nvme-4.15
> 
> for you to fetch changes up to 654b4a4acd8b52a4272114b95896e9a10d382cde:
> 
>   nvme: setup streams after initializing namespace head (2017-12-15 15:18:07 
> +0100)
> 
> 
> David Disseldorp (1):
>   nvme: set discard_alignment to zero
> 
> James Smart (1):
>   nvme-fc: remove double put reference if admin connect fails
> 
> Keith Busch (2):
>   nvme: check hw sectors before setting chunk sectors
>   nvme: setup streams after initializing namespace head
> 
> Ming Lei (1):
>   nvme: call blk_integrity_unregister after queue is cleaned up

Thanks, looks fine, pulled for 4.15.

-- 
Jens Axboe

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

[Theodore Ts'o]

On Fri, Dec 15, 2017 at 05:39:25PM +0900, Byungchul Park wrote:
> 
> All locks should belong to one class if each path of acquisition
> can be switchable each other within the class at any time.
> Otherwise, they should belong to a different class.

OK, so let's go back to my case of a Network Block Device with a local
file system mounted on it, which is then exported via NFS.

So an incoming TCP packet can go into the NFS server subsystem, then
be processed by local disk file system, which then does an I/O
operation to the NBD device, which results in an TCP packet being sent
out.  Then the response will come back over TCP, into the NBD block
layer, then into the local disk file system, and that will result in
an outgoing response to the TCP connection for the NFS protocol.

In order to avoid cross release problems, all locks associated with
the incoming TCP connection will need to be classified as belonging to
a different class as the outgoing TCP connection.  Correct?  One
solution might be to put every single TCP connection into a separate
class --- but that will explode the number of lock classes which
Lockdep will need to track, and there is a limited number of lock
classes (set at compile time) that Lockdep can track.  So if that
doesn't work, we will have to put something ugly which manually makes
certain TCP connections "magic" and require them to be put into a
separate class than all other TCP connections, which will get
collapsed into a single class.  Basically, any TCP connection which is
either originated by the kernel, or passed in from userspace into the
kernel and used by some kernel subsystem, will have to be assigned its
own lockdep class.

If the TCP connection gets closed, we don't need to track that lockdep
class any more.  (Or if a device mapper device is torn down, we
similarly don't need any unique lockdep classes created for that
device mapper device.)  Is there a way to tell lockdep that a set of
lockdep classes can be released so we can recover the kernel memory to
be used to track some new TCP connection or some new device mapper
device?

- Ted

[PATCH] Pass the task_struct explicitly to delayacct_blkio_end

[Josh Snyder]

Before e33a9bba85a8, delayacct_blkio_end was called after
context-switching into the task which completed I/O. This resulted in
double counting: the task would account a delay both waiting for I/O and
for time spent in the runqueue.

With e33a9bba85a8, delayacct_blkio_end is called by try_to_wake_up. In
ttwu, we have not yet context-switched. This is more correct, in that the
delay accounting ends when the I/O is complete. But delayacct_blkio_end
relies upon `get_current()`, and we have not yet context-switched into the
task whose I/O completed. This results in the wrong task having its delay
accounting statistics updated.

Instead of doing that, pass the task_struct being woken to
delayacct_blkio_end, so that it can update the statistics of the correct
task_struct.
---
 include/linux/delayacct.h |  8 
 kernel/delayacct.c| 42 ++
 kernel/sched/core.c   |  6 +++---
 3 files changed, 33 insertions(+), 23 deletions(-)

diff --git a/include/linux/delayacct.h b/include/linux/delayacct.h
index 4178d24..f2ad868 100644
--- a/include/linux/delayacct.h
+++ b/include/linux/delayacct.h
@@ -71,7 +71,7 @@ extern void delayacct_init(void);
 extern void __delayacct_tsk_init(struct task_struct *);
 extern void __delayacct_tsk_exit(struct task_struct *);
 extern void __delayacct_blkio_start(void);
-extern void __delayacct_blkio_end(void);
+extern void __delayacct_blkio_end(struct task_struct *);
 extern int __delayacct_add_tsk(struct taskstats *, struct task_struct *);
 extern __u64 __delayacct_blkio_ticks(struct task_struct *);
 extern void __delayacct_freepages_start(void);
@@ -122,10 +122,10 @@ static inline void delayacct_blkio_start(void)
__delayacct_blkio_start();
 }
 
-static inline void delayacct_blkio_end(void)
+static inline void delayacct_blkio_end(struct task_struct *p)
 {
if (current->delays)
-   __delayacct_blkio_end();
+   __delayacct_blkio_end(p);
delayacct_clear_flag(DELAYACCT_PF_BLKIO);
 }
 
@@ -169,7 +169,7 @@ static inline void delayacct_tsk_free(struct task_struct 
*tsk)
 {}
 static inline void delayacct_blkio_start(void)
 {}
-static inline void delayacct_blkio_end(void)
+static inline void delayacct_blkio_end(struct task_struct *p)
 {}
 static inline int delayacct_add_tsk(struct taskstats *d,
struct task_struct *tsk)
diff --git a/kernel/delayacct.c b/kernel/delayacct.c
index 4a1c334..e2ec808 100644
--- a/kernel/delayacct.c
+++ b/kernel/delayacct.c
@@ -51,16 +51,16 @@ void __delayacct_tsk_init(struct task_struct *tsk)
  * Finish delay accounting for a statistic using its timestamps (@start),
  * accumalator (@total) and @count
  */
-static void delayacct_end(u64 *start, u64 *total, u32 *count)
+static void delayacct_end(spinlock_t *lock, u64 *start, u64 *total, u32 *count)
 {
s64 ns = ktime_get_ns() - *start;
unsigned long flags;
 
if (ns > 0) {
-   spin_lock_irqsave(>delays->lock, flags);
+   spin_lock_irqsave(lock, flags);
*total += ns;
(*count)++;
-   spin_unlock_irqrestore(>delays->lock, flags);
+   spin_unlock_irqrestore(lock, flags);
}
 }
 
@@ -69,17 +69,25 @@ void __delayacct_blkio_start(void)
current->delays->blkio_start = ktime_get_ns();
 }
 
-void __delayacct_blkio_end(void)
+/*
+ * We cannot rely on the `current` macro, as we haven't yet switched back to
+ * the process being woken.
+ */
+void __delayacct_blkio_end(struct task_struct *p)
 {
-   if (current->delays->flags & DELAYACCT_PF_SWAPIN)
-   /* Swapin block I/O */
-   delayacct_end(>delays->blkio_start,
-   >delays->swapin_delay,
-   >delays->swapin_count);
-   else/* Other block I/O */
-   delayacct_end(>delays->blkio_start,
-   >delays->blkio_delay,
-   >delays->blkio_count);
+   struct task_delay_info * delays = p->delays;
+   u64 *total;
+   u32 *count;
+
+   if (p->delays->flags & DELAYACCT_PF_SWAPIN) {
+   total = >swapin_delay;
+   count = >swapin_count;
+   } else {
+   total = >blkio_delay;
+   count = >blkio_count;
+   }
+
+   delayacct_end(>lock, >blkio_start, total, count);
 }
 
 int __delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
@@ -153,8 +161,10 @@ void __delayacct_freepages_start(void)
 
 void __delayacct_freepages_end(void)
 {
-   delayacct_end(>delays->freepages_start,
-   >delays->freepages_delay,
-   >delays->freepages_count);
+   delayacct_end(
+   >delays->lock,
+   >delays->freepages_start,
+   >delays->freepages_delay,
+   >delays->freepages_count);
 }
 
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 75554f3..deda018 

Re: [PATCH V2 0/2] block: fix queue freeze and cleanup

[Bart Van Assche]

On Fri, 2017-12-15 at 15:58 +0800, chenxiang (M) wrote:
> I tested v2 of this series based on Martin's for-4.16 SCSI tree which 
> branch error handler issue "scsi: core: Ensure that the
> SCSI error handler gets woken up" is merged. And system is still hung 
> after repeat my testcase.

Hello chenxiang,

Thanks for having run this test and for having shared the results. I will see
whether I can reproduce this on my test setup.

Bart.

Re: Regression with a0747a859ef6 ("bdi: add error handle for bdi_debug_register")

[Bruno Wolff III]

On Fri, Dec 15, 2017 at 09:18:56 -0800,
 Laura Abbott  wrote:


You can see the trees Fedora produces at 
https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git
which includes the configs (you want to look at the ones withtout - debug)


Thanks. I found it a little while ago and am already doing a test build 
without weiping's test patch to see if that kernel provides what he(?) 
needs. Doing a rebuild with the test patch will go pretty quickly. So 
if I get the message with device_add_disk from these kernels, I should 
be able to get the information this afternoon. If there is some other 
reason I don't get that when I do the builds, I'm probably not going to be 
able to figure it out and get a build done before I leave. I don't live 
close enough to the office that I'm going to want to drive in just to 
be able to do a reboot test. (And my hardware at home does exhibit the 
problem.)


If you have some other idea about why I might not be seeing the 
device_add_disk message, I'd be interested in hearing it.

Re: Regression with a0747a859ef6 ("bdi: add error handle for bdi_debug_register")

[Laura Abbott]

On 12/15/2017 08:30 AM, Bruno Wolff III wrote:

On Fri, Dec 15, 2017 at 22:02:20 +0800,
  weiping zhang  wrote:


Yes, please help reproduce this issue include my debug patch. Reproduce means
we can see WARN_ON in device_add_disk caused by failure of bdi_register_owner.


I'm not sure why yet, but I'm only getting the warning message you want with 
Fedora kernels, not the ones I am building (with or without your test patch). 
I'll attach a debug config file if you want to look there. But in theory that 
should be essentially what Fedora is using for theirs. They probably have some 
out of tree patches they are applying, but I wouldn't expect those to make a 
difference here. I think they now have a tree somewhere that I can try to build 
from that has their patches applied to the upstream kernel and if I can find it 
I will try building it just to test this out.

I only have about 6 hours of physical access to the machine exhibiting the 
problem, and after that I won't be able to do test boots until Monday.



You can see the trees Fedora produces at 
https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git
which includes the configs (you want to look at the ones withtout - debug)

[GIT PULLL] nvme fixes for Linux 4.15

[Christoph Hellwig]

Hi Jens,

below are a couple important one or two liners fixes for Linux 4.15:


The following changes since commit fcf38cdf332a81b20a59e3ebaea81f6b316bbe0c:

  kyber: fix another domain token wait queue hang (2017-12-06 12:33:07 -0700)

are available in the git repository at:

  git://git.infradead.org/nvme.git nvme-4.15

for you to fetch changes up to 654b4a4acd8b52a4272114b95896e9a10d382cde:

  nvme: setup streams after initializing namespace head (2017-12-15 15:18:07 
+0100)


David Disseldorp (1):
  nvme: set discard_alignment to zero

James Smart (1):
  nvme-fc: remove double put reference if admin connect fails

Keith Busch (2):
  nvme: check hw sectors before setting chunk sectors
  nvme: setup streams after initializing namespace head

Ming Lei (1):
  nvme: call blk_integrity_unregister after queue is cleaned up

 drivers/nvme/host/core.c | 11 ++-
 drivers/nvme/host/fc.c   |  1 -
 2 files changed, 6 insertions(+), 6 deletions(-)

Re: Regression with a0747a859ef6 ("bdi: add error handle for bdi_debug_register")

[Bruno Wolff III]

On Fri, Dec 15, 2017 at 22:02:20 +0800,
 weiping zhang  wrote:


Yes, please help reproduce this issue include my debug patch. Reproduce means
we can see WARN_ON in device_add_disk caused by failure of bdi_register_owner.


I'm not sure why yet, but I'm only getting the warning message you want 
with Fedora kernels, not the ones I am building (with or without your test 
patch). I'll attach a debug config file if you want to look there. But in 
theory that should be essentially what Fedora is using for theirs. They 
probably have some out of tree patches they are applying, but I wouldn't 
expect those to make a difference here. I think they now have a tree 
somewhere that I can try to build from that has their patches applied 
to the upstream kernel and if I can find it I will try building it just 
to test this out.


I only have about 6 hours of physical access to the machine exhibiting 
the problem, and after that I won't be able to do test boots until Monday.

Re: [PATCH 2/6] blk-mq: replace timeout synchronization with a RCU and generation based scheme

[jianchao.wang]

On 12/15/2017 03:31 PM, Peter Zijlstra wrote:
> On Fri, Dec 15, 2017 at 10:12:50AM +0800, jianchao.wang wrote:
>>> That only makes it a little better:
>>>
>>> Task-A  Worker
>>>
>>> write_seqcount_begin()
>>> blk_mq_rw_update_state(rq, IN_FLIGHT)
>>> blk_add_timer(rq)
>>> 
>>> schedule_work()
>>> 
>>> 
>>> read_seqcount_begin()
>>> while(seq & 1)
>>> cpu_relax();
>>>
>> Hi Peter
>>
>> The current seqcount read side is as below:
>>  do {
>>  start = read_seqcount_begin(>gstate_seq);
> 
> 
> static inline unsigned read_seqcount_begin(const seqcount_t *s)
> {
>   seqcount_lockdep_reader_access(s);
>   return raw_read_seqcount_begin(s);
> }
> 
> static inline unsigned raw_read_seqcount_begin(const seqcount_t *s)
> {
>   unsigned ret = __read_seqcount_begin(s);
>   smp_rmb();
>   return ret;
> }
> 
> static inline unsigned __read_seqcount_begin(const seqcount_t *s)
> {
>   unsigned ret;
> 
> repeat:
>   ret = READ_ONCE(s->sequence);
>   if (unlikely(ret & 1)) {
>   cpu_relax();
>   goto repeat;
>   }
>   return ret;
> }
> 
Really thanks for kindly pointing out.
jianchao

Re: Regression with a0747a859ef6 ("bdi: add error handle for bdi_debug_register")

[weiping zhang]

2017-12-15 19:10 GMT+08:00 Bruno Wolff III :
> On Fri, Dec 15, 2017 at 10:04:32 +0800,
>  weiping zhang  wrote:
>>
>> I just want to know WARN_ON WHAT in device_add_disk,
>> if bdi_register_owner return error code, it may fail at any step of
>> following:
>
>
> Was that output in the original boot log? I didn't see anything there that
> had the string WARN_ON. The first log was from a Fedora kernel. The second
Sorry to let you confuse, WARN_ON means we catch log as following:
WARNING: CPU: 3 PID: 3486 at block/genhd.c:680 device_add_disk+0x3d9/0x460

> from a kernel I built. I used a Fedora config though. The config was
> probably from one of their nodebug kernels, I could build another one using
> a config from a debug kernel. Would that likely provide what you are looking
> for?

Yes, please help reproduce this issue include my debug patch. Reproduce means
we can see WARN_ON in device_add_disk caused by failure of bdi_register_owner.

Re: [PATCH 2/6] blk-mq: replace timeout synchronization with a RCU and generation based scheme

[t...@kernel.org]

Hello, Peter.

On Thu, Dec 14, 2017 at 09:20:42PM +0100, Peter Zijlstra wrote:
> But now that I look at this again, TJ, why can't the below happen?
> 
>   write_seqlock_begin();
>   blk_mq_rq_update_state(rq, IN_FLIGHT);
>   blk_add_timer(rq);
>   
>   read_seqcount_begin()
>   while (seq & 1)
>   cpurelax();
>   // life-lock
>   
>   write_seqlock_end();

Ah, you're right.  For both gstate_seq and aborted_gstate_sync, we can
push all synchronization to the timeout side - ie. gstate_seq read can
yield, pause or synchronize_rcu and hte aborted_gstate_sync can
disable irq around update.

Thanks.

-- 
tejun

Re: [PATCH 2/6] blk-mq: replace timeout synchronization with a RCU and generation based scheme

[t...@kernel.org]

Hello, Bart.

On Thu, Dec 14, 2017 at 09:13:32PM +, Bart Van Assche wrote:
...
> however is called before a every use of a request. Sorry but I'm not
> enthusiast about the code in blk_rq_init() that reinitializes state
> information that should survive request reuse.

If it wasn't clear, me neither.  I think what'd be better is making
those paths use the usual request allocation path instead of custom
one but for flush handling, that's not gonna be trivial, so let's deal
with that later.

Thanks.

-- 
tejun

Re: Regression with a0747a859ef6 ("bdi: add error handle for bdi_debug_register")

[Bruno Wolff III]

On Fri, Dec 15, 2017 at 10:04:32 +0800,
 weiping zhang  wrote:

I just want to know WARN_ON WHAT in device_add_disk,
if bdi_register_owner return error code, it may fail at any step of following:


Was that output in the original boot log? I didn't see anything there 
that had the string WARN_ON. The first log was from a Fedora kernel. The 
second from a kernel I built. I used a Fedora config though. The config 
was probably from one of their nodebug kernels, I could build another 
one using a config from a debug kernel. Would that likely provide what 
you are looking for?

Re: [PATCH] locking/lockdep: Remove the cross-release locking checks

[Byungchul Park]

On Fri, Dec 15, 2017 at 3:24 PM, Theodore Ts'o  wrote:
> seems that lock classification as a solution to cross-release false
> positives seems unlikely:

For this, let me explain more.

For example, either to use cross-release or to consider
wait_for_completion() in submit_bio_wait() manually using
lock_acquire() someday, classifying locks or waiters precisely
is needed.

All locks should belong to one class if each path of acquisition
can be switchable each other within the class at any time.
Otherwise, they should belong to a different class.

Even though they are different classes but belong to one class
roughly, no problem comes into view unless they are connected
each other via extra dependency chains. But, once they get
connected, we can see problems by the wrong classification.
That can happen even w/o cross-release.

Of course, as you pointed out, cross-release generates many
chains between classes, assuming all classes are well-
classified. But, practically well-classifying is not an easy work.

So that's why I suggested the way since anyway that's better
than removing all. If that's allowed, I can invalidate those waiters,
using e.g. completion_init_nomap().

>From: Byungchul Park 
>Date: Fri, 8 Dec 2017 18:27:45 +0900
>Subject: Re: [PATCH v4 72/73] xfs: Convert mru cache to XArray
>
>1) Firstly, it's hard to assign lock classes *properly*. By
>default, it relies on the caller site of lockdep_init_map(),
>but we need to assign another class manually, where ordering
>rules are complicated so cannot rely on the caller site. That
>*only* can be done by experts of the subsystem.
>
> - Ted



-- 
Thanks,
Byungchul

Re: [PATCH V2 0/2] block: fix queue freeze and cleanup

[chenxiang (M)]

在 2017/12/14 5:53, Bart Van Assche 写道:

On Wed, 2017-11-29 at 10:57 +0800, chenxiang (M) wrote:

I applied this v2 patchset to kernel 4.15-rc1, running fio on a SATA
disk, then disable the disk with sysfs interface
(echo 0 > /sys/class/sas_phy/phy-1:0:1/enable), and find system is hung.
But with v1 patch, it doesn't has this issue.

Hello chenxiang,

Would it be possible to repeat your test with v2 of this series and Martin's
for-4.16 SCSI tree merged into your kernel test tree? Martin's tree is available
at 
https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/log/?h=4.16/scsi-queue.
The following patch in that tree fixes a race condition that sometimes caused
the SCSI error handler not to be woken up:
https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=4.16/scsi-queue=3bd6f43f5cb3714f70c591514f344389df593501


Hi Bart, sorry to reply this email late, and i just noticed this email 
after John's remind yesterday.
I tested v2 of this series based on Martin's for-4.16 SCSI tree which 
branch error handler issue "scsi: core: Ensure that the
SCSI error handler gets woken up" is merged. And system is still hung 
after repeat my testcase.

Log is as follows:

estuary:/$ fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=re
ad -ioengine=psync -bs=4k -numjobs=64 -runtime=300 -group_reporting 
-name=mytes

t
mytest: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1
...
fio-2.1.11
Starting 64 threads
[   45.794460] random: crng init done
[   54.369720] hisi_sas_v2_hw HISI0162:01: erroneous completion iptt=1 
task=8017cf471400 CQ hdr: 0x1103 0x1 0x0 0x0 Error info: 0x0 0x200 
0x0 0x0
[   54.382869] sas: smp_execute_task_sg: task to dev 500e004aaa1f 
response: 0x0 status 0x2

[   54.391339] sas: broadcast received: 0
[   54.395092] sas: REVALIDATING DOMAIN on port 0, pid:2022
[   54.400879] sas: Expander phy change count has changed
[   54.406191] sas: ex 500e004aaa1f phy1 originated BROADCAST(CHANGE)
[   54.415052] sas: done REVALIDATING DOMAIN on port 0, pid:2022, res 0x0
[   54.422057] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[   54.427248] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: 
hostbyte=0x04 driverbyte=0x00
fio: pid=2786, err=5/file:engines[   54.436147] sd 0:0:1:0: [sdb] 
Stopping disk
/sync.c:67, func=xfer, error=Inpu[   54.443187] sd 0:0:1:0: [sdb] 
Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00

t/output error
fio: pid=2772, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2729, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2727, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2787, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2740, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2775, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2774, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2768, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2754, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2732, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2756, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
fio: pid=2738, err=5/file:engines/sync.c:67, func=xfer, 
error=Input/output error
[  249.291697] INFO: task kworker/u128:1:2021 blocked for more than 120 
seconds.(1),R(11),X(1),R(3),X(1),R(1),X(2),R(10),X(2),R(3)] [50.5% done] 
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 03m:20s]

[  249.298836]   Not tainted 4.15.0-rc1-00045-ga9c054d #1
[  249.304316] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.

[  249.312138] kworker/u128:1  D0  2021  2 0x0020
[  249.317632] Workqueue: HISI0162:01_event_q sas_port_event_worker
[  249.323634] Call trace:
[  249.326077]  __switch_to+0xa0/0xb4
[  249.329475]  __schedule+0x1c4/0x704
[  249.332955]  schedule+0x34/0x94
[  249.336090]  schedule_timeout+0x1b8/0x358
[  249.340092]  wait_for_common+0xfc/0x1b0
[  249.343920]  wait_for_completion+0x14/0x1c
[  249.348011]  flush_workqueue+0xfc/0x424
[  249.351840]  sas_porte_broadcast_rcvd+0x5c/0x68
[  249.356363]  sas_port_event_worker+0x24/0x38
[  249.360626]  process_one_work+0x128/0x2cc
[  249.364627]  worker_thread+0x14c/0x408
[  249.368369]  kthread+0xfc/0x128
[  249.371499]  ret_from_fork+0x10/0x18
[  249.375068] INFO: task kworker/u128:2:2022 blocked for more than 120 
seconds.

[  249.382194]   Not tainted 4.15.0-rc1-00045-ga9c054d #1
[  249.387671] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.

[  249.395492] kworker/u128:2  D0  2022  2 0x0020
[  249.400973] Workqueue: HISI0162:01_disco_q sas_revalidate_domain
[  249.406972] Call trace:
[  249.409412]  __switch_to+0xa0/0xb4
[  249.412806]  __schedule+0x1c4/0x704
[  249.416287]  schedule+0x34/0x94
[  249.419422]