I have the same issue: kcompactd0 regularity (about every 10 days)
blocks (iowait?) and never recovers. After a few more hours, the system
becomes more or less dead.
I never had this issue until, I would say, 2 month ago. I was running
Ubuntu 18.04 fine until then. No heavy load on the server.
# ps aux|grep compac
root 28 0.0 0.0 0 0 ? D Aug27 0:06 [kcompactd0]
btrfs-transaction seems OK though:
# ps aux|grep trans
root 471 0.0 0.0 0 0 ? S Aug27 7:01
[btrfs-transacti]
>From syslog:
Sep 8 16:36:04 nc4 kernel: [1037582.945988] INFO: task kcompactd0:28 blocked
for more than 120 seconds.
Sep 8 16:36:04 nc4 kernel: [1037582.946216] Not tainted
4.15.0-112-generic #113-Ubuntu
Sep 8 16:36:04 nc4 kernel: [1037582.946375] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 8 16:36:04 nc4 kernel: [1037582.946557] kcompactd0 D 0 28
2 0x80000000
Sep 8 16:36:04 nc4 kernel: [1037582.946562] Call Trace:
Sep 8 16:36:04 nc4 kernel: [1037582.946642] __schedule+0x24e/0x880
Sep 8 16:36:04 nc4 kernel: [1037582.946826] ? btree_releasepage+0x42/0x50
[btrfs]
Sep 8 16:36:04 nc4 kernel: [1037582.946831] schedule+0x2c/0x80
Sep 8 16:36:04 nc4 kernel: [1037582.946833] io_schedule+0x16/0x40
Sep 8 16:36:04 nc4 kernel: [1037582.946853] __lock_page+0xff/0x140
Sep 8 16:36:04 nc4 kernel: [1037582.946861] ? page_cache_tree_insert+0xe0/0xe0
Sep 8 16:36:04 nc4 kernel: [1037582.946869] migrate_pages+0x91f/0xb80
Sep 8 16:36:04 nc4 kernel: [1037582.946875] ? __ClearPageMovable+0x10/0x10
Sep 8 16:36:04 nc4 kernel: [1037582.946881] ?
isolate_freepages_block+0x3b0/0x3b0
Sep 8 16:36:04 nc4 kernel: [1037582.946884] compact_zone+0x681/0x950
Sep 8 16:36:04 nc4 kernel: [1037582.946887] kcompactd_do_work+0xfe/0x2a0
Sep 8 16:36:04 nc4 kernel: [1037582.946890] ? __switch_to_asm+0x35/0x70
Sep 8 16:36:04 nc4 kernel: [1037582.946892] ? __switch_to_asm+0x41/0x70
Sep 8 16:36:04 nc4 kernel: [1037582.946895] kcompactd+0x86/0x1c0
Sep 8 16:36:04 nc4 kernel: [1037582.946897] ? kcompactd+0x86/0x1c0
Sep 8 16:36:04 nc4 kernel: [1037582.946912] ? wait_woken+0x80/0x80
Sep 8 16:36:04 nc4 kernel: [1037582.946922] kthread+0x121/0x140
Sep 8 16:36:04 nc4 kernel: [1037582.946924] ? kcompactd_do_work+0x2a0/0x2a0
Sep 8 16:36:04 nc4 kernel: [1037582.946926] ?
kthread_create_worker_on_cpu+0x70/0x70
Sep 8 16:36:04 nc4 kernel: [1037582.946929] ret_from_fork+0x35/0x40
Some more infos (all while kcompactd is still blocked):
# uptime
17:20:25 up 12 days, 57 min, 1 user, load average: 1.48, 1.19, 1.08
# free -h
total used free shared buff/cache available
Mem: 5.8G 476M 1.6G 72M 3.8G 5.0G
Swap: 6.0G 568M 5.4G
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic
The root FS is BTRF on a LUKS encrypted volume.
# cryptsetup status sda5_crypt
/dev/mapper/sda5_crypt is active and is in use.
type: LUKS1
cipher: aes-xts-plain64
keysize: 512 bits
key location: dm-crypt
device: /dev/sda5
sector size: 512
offset: 4096 sectors
size: 1309712384 sectors
mode: read/write
flags: discards
# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/sda5_crypt nc4-vg lvm2 a-- <624.52g 0
# vgs
VG #PV #LV #SN Attr VSize VFree
nc4-vg 1 2 0 wz--n- <624.52g 0
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
root nc4-vg -wi-ao---- <618.52g
swap_1 nc4-vg -wi-ao---- 6.00g
I can still write to disk:
# time dd if=/dev/zero of=io-test bs=16k count=64k conv=fdatasync
65536+0 records in
65536+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.79033 s, 385 MB/s
real 0m2.792s
user 0m0.071s
sys 0m1.858s
# sar
11:05:01 AM CPU %user %nice %system %iowait %steal %idle
11:15:01 AM all 1.01 0.00 0.51 0.08 0.02 98.39
11:25:01 AM all 1.22 0.00 0.49 0.08 0.02 98.19
11:35:01 AM all 1.01 0.00 0.46 0.06 0.02 98.45
11:45:01 AM all 1.01 0.00 0.49 0.07 0.02 98.41
11:55:01 AM all 0.99 0.00 0.49 0.10 0.02 98.41
12:05:01 PM all 1.30 0.00 0.52 0.09 0.02 98.07
12:15:01 PM all 1.01 0.00 0.52 0.05 0.02 98.41
12:25:01 PM all 0.97 0.00 0.49 0.04 0.02 98.48
12:35:01 PM all 1.23 0.00 0.52 0.10 0.02 98.14
12:45:01 PM all 0.96 0.00 0.50 0.09 0.02 98.43
12:55:01 PM all 0.99 0.00 0.46 0.10 0.02 98.43
01:05:01 PM all 1.23 0.00 0.51 0.15 0.03 98.09
01:15:01 PM all 0.97 0.00 0.46 0.09 0.02 98.46
01:25:01 PM all 1.05 0.00 0.47 0.08 0.02 98.39
01:35:01 PM all 1.08 0.00 0.45 0.08 0.02 98.37
01:45:01 PM all 0.48 0.00 0.36 0.05 0.02 99.10
01:55:01 PM all 0.62 0.00 0.42 0.11 0.02 98.83
02:05:01 PM all 0.56 0.00 0.35 0.14 0.02 98.93
02:15:01 PM all 0.95 0.00 0.47 0.13 0.02 98.43
02:25:01 PM all 0.95 0.00 0.46 0.09 0.02 98.47
02:35:01 PM all 1.10 0.00 0.49 0.13 0.02 98.26
02:45:01 PM all 1.18 0.00 0.60 0.09 0.02 98.11
02:55:01 PM all 1.23 0.00 0.59 0.09 0.03 98.06
03:05:01 PM all 2.38 0.00 0.78 0.13 0.03 96.69
03:15:01 PM all 1.00 0.00 0.49 0.12 0.02 98.37
03:25:01 PM all 1.61 0.00 0.68 0.10 0.02 97.60
03:35:01 PM all 1.33 0.00 0.56 0.07 0.03 98.01
03:45:01 PM all 1.05 0.00 0.48 0.11 0.02 98.34
03:55:01 PM all 1.03 0.00 0.51 0.10 0.02 98.35
04:05:01 PM all 1.31 0.00 0.52 0.10 0.02 98.06
04:15:01 PM all 0.97 0.00 0.50 0.10 0.02 98.40
04:25:01 PM all 1.18 0.00 0.56 0.13 0.02 98.11
04:35:01 PM all 1.23 0.00 0.52 10.06 0.02 88.17
<-- trouble starts!
04:45:01 PM all 1.42 0.00 0.89 49.33 0.02 48.33
04:55:01 PM all 1.28 0.00 0.74 49.12 0.02 48.84
05:05:01 PM all 1.58 0.00 2.48 50.41 0.04 45.49
05:15:01 PM all 0.86 0.00 0.39 49.43 0.02 49.29
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1891773
Title:
kcompactd0 and btrfs-transaction keep deadlocking with each other
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891773/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
