Hello,after enabling the writeback feature in zram, I encountered the kernel bug below with heavy swap utilization. There is one specific workload that triggers the bug reliably and that is running Windows in KVM while overcommitting memory. The Windows VMs would fill all allocated memory with zero pages while booting. A few seconds after the host hits zram swap, the console on the host is flooded with the bug message. A few more seconds later I also encountered filesystem errors on the host causing the root filesystem to be mounted read-only. The filesystem errors do not occur when leaving RAM available for the host OS by limiting physical memory of the QEMU processes via cgroups.
I started three KVM instances with the following commands in my tests. Any Windows ISO or disk image can be used. Less instances and smaller allocated memory will also trigger the bug as long as swapping occurs. The type of writeback device does not seem to matter. I have tried a SATA SSD and an NVMe Optane drive so far. My test machine has 256 GB of RAM and one CPU. I saw the same behavior on another machine with two CPUs and 128 GB of RAM.
The bug does not occur when using zram as swap without "backing_dev" being set, but I had even more severe problems when running the same test on Ubuntu Kernels 4.15 and 4.17. Regardless of the writeback feature being used or not, the host would eventually lock up entirely when swap is in use on zram. The lockups may not be related directly to zram though and were apparently fixed in 4.18. I had absolutely no problems on Ubuntu Kernel 4.13 either, before the writeback feature was introduced.
Thank you for your attention. -- commands used: modprobe zram echo 1 > /sys/block/zram0/reset echo lz4 > /sys/block/zram0/comp_algorithm echo /dev/nvme0n1 > /sys/block/zram0/backing_dev echo 256G > /sys/block/zram0/disksize mkswap /dev/zram0 swapon /dev/zram0 kvm -nographic -smp 20 -m 131072 -cdrom winpe.iso -- log message: BUG: Bad page state in process qemu-system-x86 pfn:3dfab21 page:ffffdfb137eac840 count:0 mapcount:0 mapping:0000000000000000 index:0x1 flags: 0x17fffc000000008(uptodate) raw: 017fffc000000008 dead000000000100 dead000000000200 0000000000000000 raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000 page dumped because: PAGE_FLAGS_CHECK_AT_PREP flag set bad because of flags: 0x8(uptodate)Modules linked in: lz4 lz4_compress zram zsmalloc intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel bin fmt_misc pcbc aesni_intel aes_x86_64 crypto_simd cryptd iTCO_wdt glue_helper iTCO_vendor_support intel_cstate lpc_ich mei_me intel_uncore intel_rapl_perf pcspkr joydev sg mfd_core ioatdma mei wmi evdev ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad button ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto hid_generic usbhid hid sd_mod xhci_pci ehci_pci ahci libahci xhci_hcd ehci_hcd libata igb i2c_algo_bit crc32c_intel scsi_mod i2c_i8
01 dca usbcoreCPU: 4 PID: 1039 Comm: qemu-system-x86 Tainted: G B 4.18.0-rc5+ #1
Hardware name: Supermicro Super Server/X10SRL-F, BIOS 2.0b 05/02/2017 Call Trace: dump_stack+0x5c/0x7b bad_page+0xba/0x120 get_page_from_freelist+0x1016/0x1250 __alloc_pages_nodemask+0xfa/0x250 alloc_pages_vma+0x7c/0x1c0 do_swap_page+0x347/0x920 ? __update_load_avg_se.isra.38+0x1eb/0x1f0 ? cpumask_next_wrap+0x3d/0x60 __handle_mm_fault+0x7b4/0x1110 ? update_load_avg+0x5ea/0x720 handle_mm_fault+0xfc/0x1f0 __get_user_pages+0x12f/0x690 get_user_pages_unlocked+0x148/0x1f0 __gfn_to_pfn_memslot+0xff/0x3c0 [kvm] try_async_pf+0x87/0x230 [kvm] tdp_page_fault+0x132/0x290 [kvm] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] kvm_mmu_page_fault+0x74/0x570 [kvm] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmexit_fill_RSB+0x18/0x30 [kvm_intel] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmexit_fill_RSB+0x18/0x30 [kvm_intel] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmexit_fill_RSB+0x18/0x30 [kvm_intel] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmexit_fill_RSB+0x18/0x30 [kvm_intel] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmexit_fill_RSB+0x18/0x30 [kvm_intel] ? vmexit_fill_RSB+0xc/0x30 [kvm_intel] ? vmx_vcpu_run+0x375/0x620 [kvm_intel] kvm_arch_vcpu_ioctl_run+0x9b3/0x1990 [kvm] ? __update_load_avg_se.isra.38+0x1eb/0x1f0 ? kvm_vcpu_ioctl+0x388/0x5d0 [kvm] kvm_vcpu_ioctl+0x388/0x5d0 [kvm] ? __switch_to+0x395/0x450 ? __switch_to+0x395/0x450 do_vfs_ioctl+0xa2/0x630 ? __schedule+0x3fd/0x890 ksys_ioctl+0x70/0x80 ? exit_to_usermode_loop+0xca/0xf0 __x64_sys_ioctl+0x16/0x20 do_syscall_64+0x55/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fb30361add7Code: 00 00 00 48 8b 05 c1 80 2b 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 91 80 2b 00 f7 d8 64 89 01 48
RSP: 002b:00007fb2e97f98b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fb30361add7 RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000015 RBP: 00005652b984e0f0 R08: 00005652b7d513d0 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fb308c66000 R14: 0000000000000000 R15: 00005652b984e0f0 -- ver_linux: Debian 9.5 with Kernel 4.18.0-rc5+ GNU C 6.3.0 GNU Make 4.1 Binutils 2.28 Util-linux 2.29.2 Mount 2.29.2 Module-init-tools 23 E2fsprogs 1.43.4 Linux C Library 2.24 Dynamic linker (ldd) 2.24 Linux C++ Library 6.0.22 Procps 3.3.12 Kbd 2.0.3 Console-tools 2.0.3 Sh-utils 8.26 Udev 232 -- cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz stepping : 1 microcode : 0xb000021 cpu MHz : 1200.632 cache size : 25600 KB physical id : 0 siblings : 20 core id : 0 cpu cores : 10 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yesflags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass bogomips : 4400.00 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: -- Kind regards, Tino Lehnig
config-4.18.0-rc5+.gz
Description: application/gzip
