Public bug reported:
We see an issue where an IO thread of tgtd violates the hung task timeout and
vmstat reports >20% wait on IO while we are not seeing a issue with the
underlying hardware (iowait and svc time) inside IO stat.
We already limited the kernel parameters
vm.dirty_background_ratio=5
vm.dirty_ratio=10
to reduce the impact of massive dirty blocks hitting the storage which
is still based on spinning rust in a RAID 10 array.
This is based on kernel 3.13.0-96-generic with the deadline IO scheduler and we
got the following stack trace:
[25300644.048029] INFO: task tgtd:27827 blocked for more than 300 seconds.
[25300644.048065] Not tainted 3.13.0-96-generic #143-Ubuntu
[25300644.048100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[25300644.048166] tgtd D ffff88103fc13180 0 27827 1
0x00000080
[25300644.048168] ffff88019cc61c78 0000000000000086 ffff880125791800
0000000000013180
[25300644.048171] ffff88019cc61fd8 0000000000013180 ffff880125791800
ffff88103fc13a18
[25300644.048173] ffff88207ffef6e8 ffff88019cc61d00 0000000000000002
ffffffff81151e90
[25300644.048176] Call Trace:
[25300644.048178] [<ffffffff81151e90>] ? wait_on_page_read+0x60/0x60
[25300644.048190] [<ffffffff8172e71d>] io_schedule+0x9d/0x130
[25300644.048192] [<ffffffff81151e9e>] sleep_on_page+0xe/0x20
[25300644.048194] [<ffffffff8172ec98>] __wait_on_bit_lock+0x48/0xb0
[25300644.048196] [<ffffffff81151faa>] __lock_page+0x6a/0x70
[25300644.048198] [<ffffffff810adb10>] ? autoremove_wake_function+0x40/0x40
[25300644.048200] [<ffffffff8115d1c8>] write_cache_pages+0x2f8/0x4b0
[25300644.048202] [<ffffffff8115c740>] ? global_dirtyable_memory+0x50/0x50
[25300644.048205] [<ffffffff810dae48>] ? get_futex_key+0x1d8/0x2c0
[25300644.048207] [<ffffffff8107abfe>] ? send_signal+0x3e/0x80
[25300644.048209] [<ffffffff8115d3c0>] generic_writepages+0x40/0x60
[25300644.048212] [<ffffffff8115e58e>] do_writepages+0x1e/0x40
[25300644.048213] [<ffffffff81153559>] __filemap_fdatawrite_range+0x59/0x60
[25300644.048215] [<ffffffff8115365a>] filemap_write_and_wait_range+0x2a/0x70
[25300644.048217] [<ffffffff811fa79b>] blkdev_fsync+0x1b/0x50
[25300644.048219] [<ffffffff811f1841>] do_fsync+0x51/0x80
[25300644.048221] [<ffffffff811f1af3>] SyS_fdatasync+0x13/0x20
[25300644.048223] [<ffffffff8173aadd>] system_call_fastpath+0x1a/0x1f
UPTIME report
16:30:01 up 292 days, 22:34, 0 users, load average: 9.89, 8.58, 5.33
FREE report
total used free shared buffers cached
Mem: 131777272 116935780 14841492 828 110629740 1123368
-/+ buffers/cache: 5182672 126594600
Swap: 7999484 9004 7990480
VMSTAT report
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 10 8 14493 108036 1097 0 0 7 227 0 0 0 0 100 0 0
0 10 8 14494 108036 1097 0 0 0 0 690 1149 0 0 78 22 0
0 10 8 14494 108036 1097 0 0 0 8 736 1442 0 0 78 22 0
IOSTAT report
Linux 3.13.0-96-generic (cinder18) 07/19/2017 _x86_64_ (32
CPU)
07/19/2017 04:30:03 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.35 0.00 0.08 0.07 0.00 99.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 3.97 0.40 10.89 1.75 63.16 11.51
0.00 0.16 0.13 0.16 0.01 0.01
sdb 51.40 107.25 2.20 15.89 215.29 7198.82 819.79
0.07 3.89 3.26 3.97 0.60 1.08
sdc 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.11 0.11 0.00 0.11 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 11.76 0.66 14.98 0.17 0.00
dm-1 0.00 0.00 0.01 14.86 0.19 63.13 8.52
0.00 0.15 3.10 0.15 0.00 0.01
dm-2 0.00 0.00 0.00 0.00 0.03 0.02 13.67
0.00 1.07 1.66 0.69 0.47 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.51 3.51 4.00 3.51 0.00
dm-10 0.00 0.00 0.01 0.00 0.02 0.00 8.00
0.00 1.76 1.76 0.00 0.41 0.00
dm-28 0.00 0.00 0.00 0.02 0.00 0.06 8.00
0.00 1.39 5.76 1.20 0.06 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 3.62 9.63 2.97 0.02 0.00
dm-30 0.00 0.00 0.00 0.01 0.01 0.03 8.00
0.00 2.99 2.84 3.03 0.03 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.04 0.04 0.00 0.04 0.00
dm-25 0.00 0.00 0.00 0.25 0.00 0.99 8.00
0.00 6.27 3.23 6.27 0.00 0.00
dm-35 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.47 2.05 2.50 0.01 0.00
dm-36 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.98 2.69 1.71 0.09 0.00
dm-37 0.00 0.00 0.10 0.69 0.42 2.75 8.00
0.00 4.26 0.63 4.81 0.00 0.00
dm-38 0.00 0.00 0.00 0.01 0.01 0.85 188.30
0.00 2.40 4.21 1.99 0.89 0.00
dm-39 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.19 3.20 0.00 0.60 0.00
dm-40 0.00 0.00 0.02 0.00 0.08 0.00 8.00
0.00 0.20 0.20 0.00 0.01 0.00
dm-59 0.00 0.00 0.06 2.41 0.23 9.62 8.00
0.01 3.63 1.07 3.69 0.01 0.00
dm-60 0.00 0.00 0.00 0.02 0.01 0.09 8.00
0.00 3.72 16.50 1.75 0.16 0.00
dm-62 0.00 0.00 0.00 0.02 0.00 0.08 8.00
0.00 2.05 7.69 1.97 1.82 0.00
dm-63 0.00 0.00 0.03 1.29 0.10 5.16 8.00
0.01 3.96 10.41 3.84 0.20 0.03
dm-65 0.00 0.00 0.06 0.90 0.22 3.61 8.00
0.01 5.40 11.23 5.04 0.12 0.01
dm-104 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.86 8.67 2.20 0.02 0.00
dm-105 0.00 0.00 0.00 0.04 0.00 0.14 8.00
0.00 1.11 8.41 1.04 0.04 0.00
dm-106 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 3.85 7.98 3.41 0.02 0.00
dm-107 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 4.83 9.85 4.29 0.02 0.00
dm-108 0.00 0.00 0.00 0.03 0.00 0.12 8.00
0.00 1.95 15.02 1.79 0.08 0.00
dm-109 0.00 0.00 0.00 0.04 0.00 0.14 8.00
0.00 2.20 11.60 2.11 0.08 0.00
dm-116 0.00 0.00 0.00 0.01 0.00 0.04 8.00
0.00 9.27 3.84 9.34 0.01 0.00
dm-117 0.00 0.00 38.85 42.29 155.40 169.18 8.00
0.03 0.35 0.14 0.54 0.01 0.07
dm-118 0.00 0.00 0.00 25.91 0.00 103.63 8.00
0.09 3.31 10.21 3.31 0.00 0.00
dm-122 0.00 0.00 0.00 0.00 0.00 1.04 1006.91
0.00 0.29 2.25 0.26 0.16 0.00
dm-126 0.00 0.00 0.00 0.00 0.00 0.41 982.51
0.00 0.34 2.11 0.26 0.16 0.00
dm-130 0.00 0.00 0.00 0.00 0.00 2.07 1013.70
0.00 0.27 3.68 0.23 0.15 0.00
dm-134 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.61 2.61 0.00 0.44 0.00
dm-115 0.00 0.00 0.01 0.03 0.03 0.13 8.00
0.00 81.51 0.78 100.26 0.19 0.00
dm-119 0.00 0.00 0.79 1.82 3.15 7.28 8.00
0.16 60.81 1.02 86.69 0.05 0.01
dm-121 0.00 0.00 2.77 1.59 11.09 6.34 8.00
0.05 11.22 16.24 2.44 0.03 0.01
dm-149 0.00 0.00 0.00 0.01 0.00 0.03 8.00
0.00 36.92 12.62 38.03 0.02 0.00
dm-150 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 10.89 11.99 10.75 0.11 0.00
dm-151 0.00 0.00 0.00 0.01 0.00 0.04 8.00
0.00 78.01 12.63 80.68 0.08 0.00
dm-152 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.22 2.90 2.16 0.03 0.00
dm-153 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 71.05 11.97 77.41 1.03 0.00
dm-154 0.00 0.00 0.03 1.29 0.13 5.14 8.00
0.08 59.27 7.21 60.58 0.03 0.00
dm-55 0.00 0.00 4.23 0.87 16.93 3.49 8.00
0.01 1.71 1.68 1.86 0.01 0.00
dm-68 0.00 0.00 0.00 0.00 0.02 0.00 8.00
0.00 15.58 18.88 0.62 1.80 0.00
dm-96 0.00 0.00 0.00 4.94 0.00 19.74 8.00
0.02 3.18 6.69 3.18 0.00 0.00
dm-97 0.00 0.00 0.00 3.95 0.00 15.80 8.00
0.04 10.61 13.14 10.61 0.00 0.00
dm-98 0.00 0.00 0.00 0.08 0.00 0.32 8.00
0.00 51.66 1.12 51.67 0.01 0.00
dm-78 0.00 0.00 0.00 0.01 0.00 0.02 8.00
0.00 1.82 5.50 1.78 1.30 0.00
dm-79 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.78 32.34 2.20 0.15 0.00
dm-80 0.00 0.00 0.00 0.35 0.02 1.41 8.00
0.00 5.41 17.35 5.28 0.16 0.01
dm-81 0.00 0.00 0.00 0.00 0.00 0.02 8.00
0.00 7.98 8.75 7.90 0.03 0.00
dm-82 0.00 0.00 0.03 2.04 0.11 8.16 8.00
0.07 35.30 62.02 34.94 0.02 0.00
dm-83 0.00 0.00 0.60 4.38 2.39 17.52 8.00
0.04 8.01 0.38 9.05 0.00 0.00
dm-87 0.00 0.00 0.00 0.23 0.00 0.92 8.00
0.00 2.25 0.83 2.25 0.01 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.75 1.75 0.00 1.75 0.00
dm-21 0.00 0.00 0.00 0.28 0.00 1.11 8.00
0.00 4.33 9.42 4.32 0.01 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 10.76 10.76 0.00 10.76 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.91 1.96 0.00 1.91 0.00
dm-75 0.00 0.00 0.00 0.16 0.00 0.65 8.00
0.00 3.74 4.58 3.73 0.14 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.81 1.81 0.00 1.81 0.00
dm-49 0.00 0.00 0.00 0.23 0.00 0.90 8.00
0.00 4.34 11.38 4.34 0.01 0.00
dm-50 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 6.68 6.68 0.00 6.68 0.00
dm-52 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.61 2.63 0.00 2.61 0.00
dm-53 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.01 2.01 0.00 2.01 0.00
dm-61 0.00 0.00 0.00 0.03 0.00 0.12 8.00
0.00 2.20 1.83 2.20 0.02 0.00
dm-51 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.99 8.71 2.39 0.02 0.00
dm-54 0.00 0.00 0.00 0.01 0.01 0.03 8.00
0.00 3.42 3.13 3.47 0.02 0.00
dm-56 0.00 0.00 0.00 0.00 0.00 0.02 8.00
0.00 3.18 9.73 2.74 0.04 0.00
dm-72 0.00 0.00 0.01 0.07 0.03 0.29 8.00
0.00 12.51 0.40 13.57 0.00 0.00
dm-70 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.09 0.91 2.12 0.01 0.00
dm-71 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.46 3.10 2.43 0.02 0.00
dm-73 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.29 0.82 2.32 0.01 0.00
dm-47 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.17 1.17 0.00 1.17 0.00
dm-48 0.00 0.00 0.00 0.01 0.00 0.02 8.00
0.00 2.01 6.15 2.01 0.01 0.00
dm-88 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 4.75 4.76 0.00 4.75 0.00
dm-89 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-90 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.96 3.97 0.00 3.96 0.00
dm-91 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.60 0.17 3.61 0.31 0.00
dm-92 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.54 0.54 0.00 0.54 0.00
dm-93 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.17 0.17 0.00 0.17 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.31 0.31 0.00 0.31 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 1.42 2.29 1.42 0.01 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.04 0.04 0.00 0.04 0.00
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.15 0.15 0.00 0.15 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 8.04 0.00 8.07 0.16 0.00
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.29 0.29 0.00 0.29 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.48 0.48 0.00 0.48 0.00
dm-12 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.99 0.22 0.99 0.00 0.00
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.32 0.32 0.00 0.32 0.00
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.26 0.26 0.00 0.26 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.33 0.33 0.00 0.33 0.00
dm-16 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.28 0.04 0.28 0.01 0.00
dm-22 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.30 0.30 0.00 0.30 0.00
dm-23 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.09 0.09 0.00 0.09 0.00
dm-31 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.32 0.31 4.00 0.32 0.00
dm-32 0.00 0.00 0.05 0.00 0.21 1.66 66.75
0.00 0.39 0.10 4.16 0.09 0.00
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Incomplete
** Tags: trusty
** Changed in: openstack-ansible
Status: New => Invalid
** Also affects: linux (Ubuntu)
Importance: Undecided
Status: New
** No longer affects: openstack-ansible
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1705349
Title:
tgtd blocked with hitting hung_task_timeout_secs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1705349/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
