** Changed in: linux (Ubuntu)
       Status: In Progress => Fix Released

You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.

  Kernel hang on drive pull caused by regression introduced by commit

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Released

Bug description:
  == SRU Justification ==
  The following commit was applied to Xenial and introduced this
  287922eb0b18 ("block: defer timeouts to a workqueue")

  This regression was introduced in mainline as of v4.5-rc1.  Bionc was
  also affected by this regression, but it already go the fix when commit
  4e9b6f20828a was applied to mainline in v4.15-rc1.

  The regression caused a kernel hang because the HPSA driver has a tendency
  to aggressively remove missing devices.

  == Fix ==
  4e9b6f20828a ("block: Fix a race between blk_cleanup_queue() and timeout 

  == Regression Potential ==
  Low.  This commit fixes a regression and has been cc'd to stable, so it
  has had addition upstream review.  This commit is already applied to
  Bionic and Cosmic.

  == Test Case ==
  A test kernel was built with this patch and tested by the original bug 
  The bug reporter states the test kernel resolved the bug.

  A bug was introduced when backporting the fix for 
http://bugs.launchpad.net/bugs/1597908. This bug exists in all Ubuntu 16.04 LTS 
4.4 kernels >= 4.4.0-36, and many other non-LTS kernels.

  This patch changes the context in which timeout work is scheduled for
  block devices in the kernel. Previously, timeout work was executed
  directly from the timer callback that fired when a deadline was met.
  After the patch, timeout work is scheduled using a background work
  queue. This means that by the time the work executes, the device queue
  which originally scheduled the work could be torn down. In order to
  prevent this, the patch takes a reference on the device queue when
  executing the timeout work.

  The problem is that the last reference to this queue can be removed
  before the timeout work can be executed. During teardown, the block
  system executes a freeze followed by a drain. The freeze drops the
  last reference on the queue. The drain tries to clean up any
  outstanding work, including timeout work. After a freeze, the timeout
  work in the background queue is unable to obtain a reference, and
  exits early without completing work. The work is now permanently stuck
  in the queue and it will never be completed. The drain in the device
  teardown path spins indefinitely.

  The bug manifests as a hang that looks like this:
  [<ffffffff81829f15>] schedule+0x35/0x80
  [<ffffffffc014aea9>] hpsa_scan_start+0x109/0x140 [hpsa]
  [<ffffffff810c3cb0>] ? wake_atomic_t_function+0x60/0x60
  [<ffffffffc014b602>] hpsa_rescan_ctlr_worker+0x1d2/0x652 [hpsa]
  [<ffffffff8109a2c5>] process_one_work+0x165/0x480
  [<ffffffff8109a62b>] worker_thread+0x4b/0x4c0
  [<ffffffff8109a5e0>] ? process_one_work+0x480/0x480
  [<ffffffff810a0808>] kthread+0xd8/0xf0
  [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0
  [<ffffffff8182e38f>] ret_from_fork+0x3f/0x70
  [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0

  The fix exists upstream. It applies, builds, and runs cleanly on Ubuntu's 
most recent 4.4 kernel.

  We hit this bug nearly 100% of the time on some of our HP hardware.
  The HPSA driver has a tendency to aggressively remove missing devices,
  so it widens the race. As a result, we've been building our own kernel
  with this patch applied. It would be really nice if we could get it
  into mainline Ubuntu.

  Let me know what additional information is needed. Thanks!

To manage notifications about this bug go to:

Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to