** Changed in: ubuntu-power-systems
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1708130

Title:
  ISST-LTE: Ubuntu16.04.03: PowerNV: 'ppc64_cpu' commands hangs while
  changing SMT value with Leaf IO and BASE tests

Status in The Ubuntu-power-systems project:
  Triaged
Status in linux package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0 - INDIRA P. JOGA <indira.pr...@in.ibm.com> - 2017-07-07 
03:56:54 ==
  Description:
  --------------
  Started Leaf IO and BASE(without smt tests)and then tried to change the SMT 
value manually where it hangs

    UBUNTU BUILD:                         4.10.0-26-generic

  Steps to re-create:
  ------------------
  > Installed latest Ubuntu160403 kernel on system lotkvm
  4.10.0-26-generic

  > Leaf microcode: KMIPP113

  > Started Leaf IO and BASE tests(without smt tests).

  root@lotkvm:/home# show.report.py
  HOSTNAME    KERNEL VERSION       DISTRO INFO    
  --------    -----------------    -----------    
  lotkvm      4.10.0-26-generic    Ubuntu 16.04.2 LTS \n \l

  ######## Current Time: Tue Jul  4 00:55:37 2017 ########
  Job-ID        FOCUS   Start-Time              Duration                Function
  ------        -----   ----------              --------                --------
  1     IO      20170704-00:44:45       0.0 hr(s) 10.0 min(s)   IO_Focus
  2     BASE    20170704-00:44:52       0.0 hr(s) 10.0 min(s)   Test

  FOCUS                 IO      BASE    SUM
  TOTAL                 76      25      101
  FAIL          0       4       4
  PASS          76      21      97
  (%)           (100%)  (84%)   (96%)

  >Now manually changed the smt value

  root@lotkvm:/home# ppc64_cpu --smt
  SMT=8
  root@lotkvm:/home# date
  Tue Jul  4 00:46:01 CDT 2017
  root@lotkvm:/home# ppc64_cpu --smt=2
  root@lotkvm:/home# ppc64_cpu --smt
  SMT=2
  root@lotkvm:/home# date
  Tue Jul  4 00:50:01 CDT 2017
  root@lotkvm:/home# ppc64_cpu --smt=4
  root@lotkvm:/home# ppc64_cpu --smt
  SMT=4
  root@lotkvm:/home# date
  Tue Jul  4 00:54:38 CDT 2017
  root@lotkvm:/home# ppc64_cpu --smt=8

  
  [ 2055.142781] INFO: task jbd2/nvme0n1p6-:22052 blocked for more than 120 
seconds.
  [ 2055.142915]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.142978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143150] INFO: task kworker/48:0H:21755 blocked for more than 120 
seconds.
  [ 2055.143226]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.143289] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143570] INFO: task kworker/u259:3:22436 blocked for more than 120 
seconds.
  [ 2055.143647]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.143709] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143953] INFO: task kworker/8:188:118516 blocked for more than 120 
seconds.
  [ 2055.144029]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144091] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144289] INFO: task mkfs.ntfs:95505 blocked for more than 120 seconds.
  [ 2055.144353]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144416] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144600] INFO: task ppc64_cpu:80305 blocked for more than 120 seconds.
  [ 2055.144665]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144727] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144872] INFO: task rm:80950 blocked for more than 120 seconds.
  [ 2055.144936]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.145133] INFO: task rm:80951 blocked for more than 120 seconds.
  [ 2055.145195]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.145257] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2175.974718] INFO: task jbd2/nvme0n1p6-:22052 blocked for more than 120 
seconds.
  [ 2175.974848]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2175.974912] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2175.975068] INFO: task kworker/48:0H:21755 blocked for more than 120 
seconds.
  [ 2175.975144]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2175.975206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.

  
  >  root@lotkvm:# ps -eaf | grep ppc*
  root      48054  12068  0 01:24 pts/0    00:00:00 grep --color=auto ppc*
  root      80305   5719  0 00:54 hvc0     00:00:00 ppc64_cpu --smt 8

  > ppc64_cpu --smt command hangs here . Not able to change the SMT
  value from 4 to 8.

  > Attached dmesg logs

  == Comment: #2 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-07-07 
13:29:41 ==
  From kernel logs
  =============

  [ 2055.142781] INFO: task jbd2/nvme0n1p6-:22052 blocked for more than 120 
seconds.
  [ 2055.142915]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.142978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143055] jbd2/nvme0n1p6- D    0 22052      2 0x00000800
  [ 2055.143059] Call Trace:
  [ 2055.143063] [c000001e1c6537f0] [c0000017dfd2206c] 0xc0000017dfd2206c 
(unreliable)
  [ 2055.143070] [c000001e1c6539c0] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.143075] [c000001e1c653a20] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.143079] [c000001e1c653b00] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.143083] [c000001e1c653b30] [c000000000495d88] 
jbd2_journal_commit_transaction+0x248/0x1e60
  [ 2055.143087] [c000001e1c653d30] [c00000000049eb50] kjournald2+0xf0/0x300
  [ 2055.143091] [c000001e1c653dc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2055.143095] [c000001e1c653e30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 2055.143150] INFO: task kworker/48:0H:21755 blocked for more than 120 
seconds.
  [ 2055.143226]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.143289] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143364] kworker/48:0H   D    0 21755      2 0x00000800
  [ 2055.143414] Workqueue: xfs-log/nvme0n1p3 xfs_log_worker [xfs]
  [ 2055.143416] Call Trace:
  [ 2055.143418] [c000001e1338b5b0] [c000001e1338b6b0] 0xc000001e1338b6b0 
(unreliable)
  [ 2055.143421] [c000001e1338b780] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.143425] [c000001e1338b7e0] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.143428] [c000001e1338b8c0] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.143431] [c000001e1338b8f0] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2055.143434] [c000001e1338b9e0] [c000000000ba054c] 
wait_for_common+0xec/0x250
  [ 2055.143438] [c000001e1338ba60] [c00000000010cc1c] flush_work+0x14c/0x2a0
  [ 2055.143477] [c000001e1338baf0] [d000000017bd2378] 
xlog_cil_force_lsn+0x98/0x300 [xfs]
  [ 2055.143516] [c000001e1338bbb0] [d000000017bcf41c] 
_xfs_log_force+0xbc/0x3f0 [xfs]
  [ 2055.143555] [c000001e1338bc50] [d000000017bcf8c8] 
xfs_log_worker+0x58/0x1a0 [xfs]
  [ 2055.143558] [c000001e1338bc90] [c00000000010e1f8] 
process_one_work+0x1e8/0x5b0
  [ 2055.143562] [c000001e1338bd20] [c00000000010e668] worker_thread+0xa8/0x660
  [ 2055.143565] [c000001e1338bdc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2055.143568] [c000001e1338be30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 2055.143570] INFO: task kworker/u259:3:22436 blocked for more than 120 
seconds.
  [ 2055.143647]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.143709] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.143784] kworker/u259:3  D    0 22436      2 0x00000800
  [ 2055.143790] Workqueue: writeback wb_workfn (flush-7:0)
  [ 2055.143792] Call Trace:
  [ 2055.143796] [c000001e133eeee0] [c0000000014e2090] 
sysctl_sched_migration_cost+0x0/0x4 (unreliable)
  [ 2055.143800] [c000001e133ef0b0] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.143803] [c000001e133ef110] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.143806] [c000001e133ef1f0] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.143809] [c000001e133ef220] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2055.143813] [c000001e133ef310] [c000000000b9eab4] 
io_schedule_timeout+0xd4/0x170
  [ 2055.143815] [c000001e133ef360] [c000000000620afc] wbt_wait+0x3ac/0x4d0
  [ 2055.143820] [c000001e133ef400] [c0000000005ecf00] 
blk_sq_make_request+0x110/0x540
  [ 2055.143823] [c000001e133ef4b0] [c0000000005da094] 
generic_make_request+0x154/0x310
  [ 2055.143826] [c000001e133ef510] [c0000000005da320] submit_bio+0xd0/0x1f0
  [ 2055.143829] [c000001e133ef5c0] [c0000000003b1460] submit_bh_wbc+0x1c0/0x220
  [ 2055.143832] [c000001e133ef610] [c0000000003b1740] 
__block_write_full_page+0x280/0x570
  [ 2055.143835] [c000001e133ef6b0] [c0000000003b6960] 
blkdev_writepage+0x30/0x50
  [ 2055.143839] [c000001e133ef6d0] [c000000000297af0] __writepage+0x40/0xb0
  [ 2055.143843] [c000001e133ef700] [c00000000029702c] 
write_cache_pages+0x25c/0x5a0
  [ 2055.143846] [c000001e133ef840] [c0000000002973d4] 
generic_writepages+0x64/0xa0
  [ 2055.143849] [c000001e133ef8a0] [c0000000003b68a4] 
blkdev_writepages+0x44/0x80
  [ 2055.143851] [c000001e133ef8c0] [c00000000029a33c] do_writepages+0x4c/0x80
  [ 2055.143855] [c000001e133ef8e0] [c0000000003a2bc0] 
__writeback_single_inode+0x70/0x510
  [ 2055.143858] [c000001e133ef940] [c0000000003a372c] 
writeback_sb_inodes+0x2cc/0x590
  [ 2055.143861] [c000001e133efa50] [c0000000003a3ad4] 
__writeback_inodes_wb+0xe4/0x150
  [ 2055.143863] [c000001e133efab0] [c0000000003a3f3c] wb_writeback+0x2fc/0x440
  [ 2055.143866] [c000001e133efb80] [c0000000003a4d78] wb_workfn+0x268/0x580
  [ 2055.143870] [c000001e133efc90] [c00000000010e1f8] 
process_one_work+0x1e8/0x5b0
  [ 2055.143873] [c000001e133efd20] [c00000000010e668] worker_thread+0xa8/0x660
  [ 2055.143876] [c000001e133efdc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2055.143879] [c000001e133efe30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 2055.143953] INFO: task kworker/8:188:118516 blocked for more than 120 
seconds.
  [ 2055.144029]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144091] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144165] kworker/8:188   D    0 118516      2 0x00000800
  [ 2055.144171] Workqueue: events vmstat_shepherd
  [ 2055.144172] Call Trace:
  [ 2055.144174] [c000001da4be3830] [c000001da4be3860] 0xc000001da4be3860 
(unreliable)
  [ 2055.144178] [c000001da4be3a00] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.144181] [c000001da4be3a60] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.144184] [c000001da4be3b40] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.144187] [c000001da4be3b70] [c000000000b9fa34] 
schedule_preempt_disabled+0x24/0x40
  [ 2055.144191] [c000001da4be3b90] [c000000000ba2338] 
__mutex_lock_slowpath+0x208/0x380
  [ 2055.144195] [c000001da4be3c10] [c0000000000e9cbc] get_online_cpus+0x5c/0xa0
  [ 2055.144197] [c000001da4be3c40] [c0000000002b867c] 
vmstat_shepherd+0x3c/0x160
  [ 2055.144201] [c000001da4be3c90] [c00000000010e1f8] 
process_one_work+0x1e8/0x5b0
  [ 2055.144204] [c000001da4be3d20] [c00000000010e668] worker_thread+0xa8/0x660
  [ 2055.144206] [c000001da4be3dc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2055.144210] [c000001da4be3e30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 2055.144289] INFO: task mkfs.ntfs:95505 blocked for more than 120 seconds.
  [ 2055.144353]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144416] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144490] mkfs.ntfs       D    0 95505  92952 0x00040000
  [ 2055.144492] Call Trace:
  [ 2055.144494] [c000000f52ec3270] [c000000f52ec32b0] 0xc000000f52ec32b0 
(unreliable)
  [ 2055.144498] [c000000f52ec3440] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.144501] [c000000f52ec34a0] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.144504] [c000000f52ec3580] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.144507] [c000000f52ec35b0] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2055.144510] [c000000f52ec36a0] [c000000000b9eab4] 
io_schedule_timeout+0xd4/0x170
  [ 2055.144513] [c000000f52ec36f0] [c000000000620afc] wbt_wait+0x3ac/0x4d0
  [ 2055.144516] [c000000f52ec3790] [c0000000005ecf00] 
blk_sq_make_request+0x110/0x540
  [ 2055.144520] [c000000f52ec3840] [c0000000005da094] 
generic_make_request+0x154/0x310
  [ 2055.144523] [c000000f52ec38a0] [c0000000005da320] submit_bio+0xd0/0x1f0
  [ 2055.144525] [c000000f52ec3950] [c0000000003b1460] submit_bh_wbc+0x1c0/0x220
  [ 2055.144528] [c000000f52ec39a0] [c0000000003b1740] 
__block_write_full_page+0x280/0x570
  [ 2055.144531] [c000000f52ec3a40] [c0000000003b6960] 
blkdev_writepage+0x30/0x50
  [ 2055.144534] [c000000f52ec3a60] [c000000000297af0] __writepage+0x40/0xb0
  [ 2055.144537] [c000000f52ec3a90] [c00000000029702c] 
write_cache_pages+0x25c/0x5a0
  [ 2055.144541] [c000000f52ec3bd0] [c0000000002973d4] 
generic_writepages+0x64/0xa0
  [ 2055.144544] [c000000f52ec3c30] [c0000000003b68a4] 
blkdev_writepages+0x44/0x80
  [ 2055.144546] [c000000f52ec3c50] [c00000000029a33c] do_writepages+0x4c/0x80
  [ 2055.144549] [c000000f52ec3c70] [c000000000286478] 
__filemap_fdatawrite_range+0x108/0x190
  [ 2055.144552] [c000000f52ec3d10] [c000000000286748] 
filemap_write_and_wait_range+0x68/0xf0
  [ 2055.144555] [c000000f52ec3d50] [c0000000003b5534] blkdev_fsync+0x34/0xa0
  [ 2055.144558] [c000000f52ec3d80] [c0000000003a9ed8] 
vfs_fsync_range+0x78/0x170
  [ 2055.144561] [c000000f52ec3dd0] [c0000000003aa06c] do_fsync+0x5c/0xb0
  [ 2055.144564] [c000000f52ec3e10] [c0000000003aa46c] SyS_fsync+0x2c/0x40
  [ 2055.144567] [c000000f52ec3e30] [c00000000000b184] system_call+0x38/0xe0
  [ 2055.144600] INFO: task ppc64_cpu:80305 blocked for more than 120 seconds.
  [ 2055.144665]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144727] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.144801] ppc64_cpu       D    0 80305   5719 0x00040000
  [ 2055.144804] Call Trace:
  [ 2055.144805] [c000001d963d7650] [c000001d963d76a0] 0xc000001d963d76a0 
(unreliable)
  [ 2055.144809] [c000001d963d7820] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.144812] [c000001d963d7880] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.144815] [c000001d963d7960] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.144818] [c000001d963d7990] [c0000000005e9f24] 
blk_mq_freeze_queue_wait+0x94/0x120
  [ 2055.144822] [c000001d963d7a00] [c0000000005ec848] 
blk_mq_queue_reinit_work+0xb8/0x180
  [ 2055.144825] [c000001d963d7a40] [c0000000005ec9f8] 
blk_mq_queue_reinit_prepare+0x88/0xa0
  [ 2055.144828] [c000001d963d7a70] [c0000000000e851c] 
cpuhp_invoke_callback+0x17c/0x600
  [ 2055.144831] [c000001d963d7ae0] [c0000000000e8bf8] 
cpuhp_up_callbacks+0x58/0x150
  [ 2055.144835] [c000001d963d7b30] [c0000000000eb444] _cpu_up+0xf4/0x1d0
  [ 2055.144838] [c000001d963d7b90] [c0000000000eb650] do_cpu_up+0x130/0x160
  [ 2055.144841] [c000001d963d7c10] [c00000000078681c] 
cpu_subsys_online+0x6c/0xf0
  [ 2055.144845] [c000001d963d7c60] [c00000000077dd64] device_online+0xb4/0x120
  [ 2055.144848] [c000001d963d7ca0] [c00000000077de84] online_store+0xb4/0xc0
  [ 2055.144850] [c000001d963d7ce0] [c000000000778b40] dev_attr_store+0x40/0x70
  [ 2055.144853] [c000001d963d7d00] [c00000000041de9c] sysfs_kf_write+0x6c/0xa0
  [ 2055.144856] [c000001d963d7d20] [c00000000041cd4c] 
kernfs_fop_write+0x17c/0x250
  [ 2055.144860] [c000001d963d7d70] [c00000000035a6e0] __vfs_write+0x40/0x80
  [ 2055.144863] [c000001d963d7d90] [c00000000035c2b4] vfs_write+0xd4/0x270
  [ 2055.144866] [c000001d963d7de0] [c00000000035df1c] SyS_write+0x6c/0x110
  [ 2055.144869] [c000001d963d7e30] [c00000000000b184] system_call+0x38/0xe0
  [ 2055.144872] INFO: task rm:80950 blocked for more than 120 seconds.
  [ 2055.144936]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.144998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.145072] rm              D    0 80950  78982 0x00040000
  [ 2055.145075] Call Trace:
  [ 2055.145077] [c000000f694234d0] [c000000f694235f0] 0xc000000f694235f0 
(unreliable)
  [ 2055.145080] [c000000f694236a0] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.145083] [c000000f69423700] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.145086] [c000000f694237e0] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.145090] [c000000f69423810] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2055.145093] [c000000f69423900] [c000000000b9eab4] 
io_schedule_timeout+0xd4/0x170
  [ 2055.145096] [c000000f69423950] [c000000000ba0224] bit_wait_io+0x34/0x90
  [ 2055.145099] [c000000f69423980] [c000000000b9fc18] __wait_on_bit+0xf8/0x170
  [ 2055.145102] [c000000f694239d0] [c000000000b9fff8] 
out_of_line_wait_on_bit+0x88/0xa0
  [ 2055.145105] [c000000f69423a50] [c000000000493710] 
do_get_write_access+0x370/0x630
  [ 2055.145108] [c000000f69423b30] [c000000000493a5c] 
jbd2_journal_get_write_access+0x8c/0xf0
  [ 2055.145112] [c000000f69423b60] [c0000000004707bc] 
__ext4_journal_get_write_access+0x8c/0xe0
  [ 2055.145115] [c000000f69423ba0] [c0000000004333ac] 
ext4_reserve_inode_write+0xcc/0x100
  [ 2055.145119] [c000000f69423bf0] [c000000000433444] 
ext4_mark_inode_dirty+0x64/0x270
  [ 2055.145121] [c000000f69423ca0] [c0000000004445d0] ext4_unlink+0x370/0x400
  [ 2055.145124] [c000000f69423d40] [c00000000036d984] vfs_unlink+0x104/0x2a0
  [ 2055.145128] [c000000f69423d90] [c000000000374ef8] do_unlinkat+0x368/0x3a0
  [ 2055.145131] [c000000f69423e30] [c00000000000b184] system_call+0x38/0xe0
  [ 2055.145133] INFO: task rm:80951 blocked for more than 120 seconds.
  [ 2055.145195]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2055.145257] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2055.145331] rm              D    0 80951 110079 0x00040000
  [ 2055.145334] Call Trace:
  [ 2055.145337] [c0000007716374d0] [c0000000014e2090] 
sysctl_sched_migration_cost+0x0/0x4 (unreliable)
  [ 2055.145340] [c0000007716376a0] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.145343] [c000000771637700] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.145346] [c0000007716377e0] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.145350] [c000000771637810] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2055.145353] [c000000771637900] [c000000000b9eab4] 
io_schedule_timeout+0xd4/0x170
  [ 2055.145356] [c000000771637950] [c000000000ba0224] bit_wait_io+0x34/0x90
  [ 2055.145359] [c000000771637980] [c000000000b9fc18] __wait_on_bit+0xf8/0x170
  [ 2055.145362] [c0000007716379d0] [c000000000b9fff8] 
out_of_line_wait_on_bit+0x88/0xa0
  [ 2055.145365] [c000000771637a50] [c000000000493710] 
do_get_write_access+0x370/0x630
  [ 2055.145368] [c000000771637b30] [c000000000493a5c] 
jbd2_journal_get_write_access+0x8c/0xf0
  [ 2055.145371] [c000000771637b60] [c0000000004707bc] 
__ext4_journal_get_write_access+0x8c/0xe0
  [ 2055.145374] [c000000771637ba0] [c0000000004333ac] 
ext4_reserve_inode_write+0xcc/0x100
  [ 2055.145377] [c000000771637bf0] [c000000000433444] 
ext4_mark_inode_dirty+0x64/0x270
  [ 2055.145380] [c000000771637ca0] [c0000000004445d0] ext4_unlink+0x370/0x400
  [ 2055.145383] [c000000771637d40] [c00000000036d984] vfs_unlink+0x104/0x2a0
  [ 2055.145386] [c000000771637d90] [c000000000374ef8] do_unlinkat+0x368/0x3a0
  [ 2055.145389] [c000000771637e30] [c00000000000b184] system_call+0x38/0xe0
  [ 2175.974718] INFO: task jbd2/nvme0n1p6-:22052 blocked for more than 120 
seconds.
  [ 2175.974848]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2175.974912] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2175.974988] jbd2/nvme0n1p6- D    0 22052      2 0x00000800
  [ 2175.974991] Call Trace:
  [ 2175.974994] [c000001e1c6537f0] [c0000017dfd2206c] 0xc0000017dfd2206c 
(unreliable)
  [ 2175.975001] [c000001e1c6539c0] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2175.975006] [c000001e1c653a20] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2175.975009] [c000001e1c653b00] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2175.975014] [c000001e1c653b30] [c000000000495d88] 
jbd2_journal_commit_transaction+0x248/0x1e60
  [ 2175.975017] [c000001e1c653d30] [c00000000049eb50] kjournald2+0xf0/0x300
  [ 2175.975022] [c000001e1c653dc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2175.975025] [c000001e1c653e30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 2175.975068] INFO: task kworker/48:0H:21755 blocked for more than 120 
seconds.
  [ 2175.975144]       Not tainted 4.10.0-26-generic #30~16.04.1-Ubuntu
  [ 2175.975206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [ 2175.975281] kworker/48:0H   D    0 21755      2 0x00000800
  [ 2175.975329] Workqueue: xfs-log/nvme0n1p3 xfs_log_worker [xfs]
  [ 2175.975331] Call Trace:
  [ 2175.975332] [c000001e1338b5b0] [c000001e1338b6b0] 0xc000001e1338b6b0 
(unreliable)
  [ 2175.975336] [c000001e1338b780] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2175.975339] [c000001e1338b7e0] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2175.975342] [c000001e1338b8c0] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2175.975346] [c000001e1338b8f0] [c000000000ba45c4] 
schedule_timeout+0x284/0x480
  [ 2175.975349] [c000001e1338b9e0] [c000000000ba054c] 
wait_for_common+0xec/0x250
  [ 2175.975353] [c000001e1338ba60] [c00000000010cc1c] flush_work+0x14c/0x2a0
  [ 2175.975392] [c000001e1338baf0] [d000000017bd2378] 
xlog_cil_force_lsn+0x98/0x300 [xfs]
  [ 2175.975431] [c000001e1338bbb0] [d000000017bcf41c] 
_xfs_log_force+0xbc/0x3f0 [xfs]
  [ 2175.975470] [c000001e1338bc50] [d000000017bcf8c8] 
xfs_log_worker+0x58/0x1a0 [xfs]
  [ 2175.975474] [c000001e1338bc90] [c00000000010e1f8] 
process_one_work+0x1e8/0x5b0
  [ 2175.975477] [c000001e1338bd20] [c00000000010e668] worker_thread+0xa8/0x660
  [ 2175.975480] [c000001e1338bdc0] [c00000000011768c] kthread+0x16c/0x1b0
  [ 2175.975483] [c000001e1338be30] [c00000000000b4e8] 
ret_from_kernel_thread+0x5c/0x74
  [ 3810.434586] Buffer I/O error on dev loop1, logical block 2399, async page 
read

  == Comment: #7 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-07-12 
05:45:57 ==
  [ 2055.144801] ppc64_cpu       D    0 80305   5719 0x00040000
  [ 2055.144804] Call Trace:
  [ 2055.144805] [c000001d963d7650] [c000001d963d76a0] 0xc000001d963d76a0 
(unreliable)
  [ 2055.144809] [c000001d963d7820] [c00000000001be70] __switch_to+0x2c0/0x450
  [ 2055.144812] [c000001d963d7880] [c000000000b9ee48] __schedule+0x2f8/0x970
  [ 2055.144815] [c000001d963d7960] [c000000000b9f50c] schedule+0x4c/0xc0
  [ 2055.144818] [c000001d963d7990] [c0000000005e9f24] 
blk_mq_freeze_queue_wait+0x94/0x120
  [ 2055.144822] [c000001d963d7a00] [c0000000005ec848] 
blk_mq_queue_reinit_work+0xb8/0x180
  [ 2055.144825] [c000001d963d7a40] [c0000000005ec9f8] 
blk_mq_queue_reinit_prepare+0x88/0xa0
  [ 2055.144828] [c000001d963d7a70] [c0000000000e851c] 
cpuhp_invoke_callback+0x17c/0x600
  [ 2055.144831] [c000001d963d7ae0] [c0000000000e8bf8] 
cpuhp_up_callbacks+0x58/0x150
  [ 2055.144835] [c000001d963d7b30] [c0000000000eb444] _cpu_up+0xf4/0x1d0
  [ 2055.144838] [c000001d963d7b90] [c0000000000eb650] do_cpu_up+0x130/0x160
  [ 2055.144841] [c000001d963d7c10] [c00000000078681c] 
cpu_subsys_online+0x6c/0xf0
  [ 2055.144845] [c000001d963d7c60] [c00000000077dd64] device_online+0xb4/0x120
  [ 2055.144848] [c000001d963d7ca0] [c00000000077de84] online_store+0xb4/0xc0
  [ 2055.144850] [c000001d963d7ce0] [c000000000778b40] dev_attr_store+0x40/0x70
  [ 2055.144853] [c000001d963d7d00] [c00000000041de9c] sysfs_kf_write+0x6c/0xa0
  [ 2055.144856] [c000001d963d7d20] [c00000000041cd4c] 
kernfs_fop_write+0x17c/0x250
  [ 2055.144860] [c000001d963d7d70] [c00000000035a6e0] __vfs_write+0x40/0x80
  [ 2055.144863] [c000001d963d7d90] [c00000000035c2b4] vfs_write+0xd4/0x270
  [ 2055.144866] [c000001d963d7de0] [c00000000035df1c] SyS_write+0x6c/0x110
  [ 2055.144869] [c000001d963d7e30] [c00000000000b184] system_call+0x38/0xe0

  ppc64_cpu process is stuck in blk_mq routines. Similar issue has also 
  reported online also as below:

  http://marc.info/?l=linux-block&m=149056984921439&w=2

  == Comment: #29 - Wen Xiong <wenxi...@us.ibm.com> - 2017-07-31 16:51:16 ==
  Paul from kernel team suggested to drop the following patch into Ubuntu. We 
have verified the patch fixed the ppc64_cpu hung issue.

  http://marc.info/?l=linux-block&m=149860870015032&w=2

  Let us know if you have any questions.

  Thanks,
  Wendy

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1708130/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to