Public bug reported: PROBLEM:
After updating to 3.2.0-30-generic on an Ubuntu 12.04 x86_64 server with 24 SATA drives and md RAID. When a hotplug event occurs for a drive (even a drive which is not part of any active md RAID set), the machine hangs in a really bad way. I can ping it, I can open a connection to port 22 but not start ssh. Console reports messages such as "BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u:12:1602]" For more details including stacktrace see http://gluster.org/pipermail/gluster-users/2012-September/011354.html This has a screenshot as attachment: http://gluster.org/pipermail/gluster-users/attachments/20120909/40d0cfb0/attachment-0001.png CAUSE AND FIX: Patch provided at http://gluster.org/pipermail/gluster-users/2012-September/011355.html It says this was a regression introduced by commit 3b661a9 "[SCSI] fix hot unplug vs async scan race" which is 1675b80 in the ubuntu-precise repository. After building a kernel with this patch applied, I found that: - hot plugging two inactive drives while I/O access is going on to the other drives is fine. The other drives in an md raid0 set continued to work without a hitch (activity was being generated by bonnie++) - Even removing those two drives while dd'ing from them was fine. I/O to the other md RAID set was also unaffected. NOTE ABOUT TESTED KERNEL: I built this test kernel from apt-get install linux-source (version 3.2.0-30.48), untarring /usr/src/linux-source-3.2.0.tar.bz2, applying the patch, and then "fakeroot make-kpkg --initrd --append-to- version=-brian-20120910 kernel-image kernel-headers" However the deb package built is "linux-image-3.2.27-brian-20120910_3.2.27-brian-20120910-10.00.Custom_amd64.deb" and uname -a also reports "3.2.27-brian-20120910" I was surprised to find my kernel called 3.2.27 instead of 3.2.0, and so I wonder if there are other changes in this kernel apart from the patch I applied. ADDITIONAL SYSTEM DETAILS: Ubuntu 12.04 x86_64 server 24 SATA drives 2 x LSI HBAs (1 x 8 port, 1 x 16 port) 03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 09:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02) # ./sas2flash -listall LSI Corporation SAS2 Flash Utility Version 12.00.00.00 (2011.11.08) Copyright (c) 2008-2011 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2008(B2) Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr ---------------------------------------------------------------------------- 0 SAS2008(B2) 12.00.00.00 0c.00.00.05 07.23.01.00 00:03:00:00 1 SAS2116_1(B1) 12.00.00.00 0c.00.00.01 07.23.01.00 00:09:00:00 Finished Processing Commands Successfully. Exiting SAS2Flash. ** Affects: linux (Ubuntu) Importance: Undecided Status: Incomplete ** Tags: precise -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1049013 Title: SATA hotplug causes I/O stack to freeze To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1049013/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
