------- Comment From [email protected] 2017-03-08 05:56 EDT------- Just tried the newest Kernel 4.4.0-66, and I'm still running into the hang. Here the final statements in /var/log/syslog (the lines, that never make it out onto the disk):
Mar 8 11:26:31 mclint multipathd[955]: mpatha: sdb - tur checker timed out Mar 8 11:26:31 mclint multipathd[955]: 8:16: reinstated Mar 8 11:26:31 mclint multipathd[955]: mpatha: sdd - tur checker timed out Mar 8 11:26:31 mclint rsyslogd-2007: action 'action 10' suspended, next retry is Wed Mar 8 11:27:01 2017 [v8.16.0 try http://www.rsyslog.com/e/2007 ] Mar 8 11:26:31 mclint multipathd[955]: 8:48: reinstated Mar 8 11:26:31 mclint multipathd[955]: mpatha: sdc - tur checker timed out Mar 8 11:26:31 mclint multipathd[955]: 8:32: reinstated Mar 8 11:26:32 mclint multipathd[955]: mpatha: sda - tur checker timed out Mar 8 11:26:32 mclint multipathd[955]: 8:0: reinstated And this here shows up on the sclp_line console: ? 961.419327! INFO: task cpuplugd:2604 blocked for more than 120 seconds. ? 961.419337! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419338! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419404! INFO: task irqbalance:2651 blocked for more than 120 seconds. ? 961.419406! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419407! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419450! INFO: task kworker/0:4:3801 blocked for more than 120 seconds. ? 961.419451! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419452! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419494! INFO: task kworker/1:1:4548 blocked for more than 120 seconds. ? 961.419495! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419496! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419539! INFO: task kworker/0:0H:20302 blocked for more than 120 seconds. ? 961.419540! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419541! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419764! INFO: task kworker/0:0:66641 blocked for more than 120 seconds. ? 961.419766! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419767! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 961.419895! INFO: task rm:81710 blocked for more than 120 seconds. ? 961.419896! Not tainted 4.4.0-66-generic #87-Ubuntu ? 961.419897! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 1081.419024! INFO: task systemd:1 blocked for more than 120 seconds. ? 1081.419033! Not tainted 4.4.0-66-generic #87-Ubuntu ? 1081.419035! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 1081.419148! INFO: task cpuplugd:2604 blocked for more than 120 seconds. ? 1081.419150! Not tainted 4.4.0-66-generic #87-Ubuntu ? 1081.419151! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ? 1081.419186! INFO: task irqbalance:2651 blocked for more than 120 seconds. ? 1081.419187! Not tainted 4.4.0-66-generic #87-Ubuntu ? 1081.419188! "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. I pulled a DASD-Dump from the system: KERNEL: /usr/lib/debug/boot/vmlinux-4.4.0-66-generic DUMPFILE: mclint_20170308_kernel_4_4_0-66_without_openafs.dump CPUS: 3 DATE: Wed Mar 8 11:37:56 2017 UPTIME: 00:25:30 LOAD AVERAGE: 12.99, 11.25, 6.55 TASKS: 422 NODENAME: mclint RELEASE: 4.4.0-66-generic VERSION: #87-Ubuntu SMP Fri Mar 3 15:32:53 UTC 2017 MACHINE: s390x (unknown Mhz) MEMORY: 7.8 GB PANIC: "" PID: 0 COMMAND: "swapper/0" TASK: bb1538 (1 of 3) [THREAD_INFO: b7c000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) INFO: no panic task found And again I see 10 multipath-Daemons in the process list, this is my typical hang scenario. crash> ps | grep multipathd 955 1 0 1e49115f0 IN 0.1 335364 8316 multipathd 971 1 0 7e8b2be0 IN 0.1 335364 8316 multipathd 972 1 0 7e8b6db0 IN 0.1 335364 8316 multipathd 977 1 1 7e8b36d8 IN 0.1 335364 8316 multipathd 978 1 0 7e8b62b8 IN 0.1 335364 8316 multipathd 979 1 2 7e8b4cc8 IN 0.1 335364 8316 multipathd 81714 1 1 7cdc8000 IN 0.1 335364 8316 multipathd 81715 1 1 7cdc95f0 IN 0.1 335364 8316 multipathd 81716 1 1 7cdcc1d0 IN 0.1 335364 8316 multipathd 81717 1 1 1e6c595f0 IN 0.1 335364 8316 multipathd I'll compress the dump and try to find ways to make it available to you ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1670634 Title: blk-mq: possible deadlock on CPU hot(un)plug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1670634/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
