Re: [Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Great, thanks! Robert Am 11.03.2016 15:01 schrieb "Seth Forshee" <seth.forshee...@canonical.com>: > On Fri, Mar 11, 2016 at 01:03:32PM -0000, Robert Doebbelin wrote: > > Thank you Seth for taking a close look at the problem and my proposed > > fix. As mentioned on the mailing list my test runs fine now with the two > > fixes. > > > > However, I prefer your fix as it prevents us from running into this > > issue again. Our test system is happily installing VMs for two hours now > > using your build. Please propose your patch. > > I'm not subscribed to fuse-devel and hadn't refreshed the mailing list > thread so I didn't realize that you had discovered that the hang was > unrelated. That's good. > > I'm happy to send the patches, I'll go ahead and send both my patch and > your iocb patch after I make sure it all applies/builds okay on 4.5. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1505948 > > Title: > Memory arena corruption with FUSE (was Memory allocation failure > crashes kernel hard, presumably related to FUSE) > > Status in linux package in Ubuntu: > Confirmed > Status in linux source package in Wily: > Confirmed > Status in linux package in Fedora: > Unknown > > Bug description: > Hello everybody, > > Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our > setup when trying to start a Qemu process on top of a fuse-based > mount. Here is an example stacktrace: > > [ 739.807817] BUG: unable to handle kernel paging request at > 8800a4104ea0 > [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 > [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 > [ 739.890418] Oops: [#1] SMP > [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre > ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa > ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi > ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter > xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit > nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack > ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi > x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul > crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul > glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost > macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac > 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek > ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 > sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure > hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core > hpsa mdio wmi > [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted > 4.2.0-040200-generic #201508301530 > [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 > [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: > 882f28c2 > [ 740.451672] RIP: 0010:[] [] > kmem_cache_alloc_trace+0x7a/0x1f0 > [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 > [ 740.518425] RAX: RBX: 00d0 RCX: > 26b3 > [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: > 882fbf407840 > [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: > e8d000200ab0 > [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: > 00d0 > [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: > 882fbf407840 > [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() > knlGS: > [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 > [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: > 001426e0 > [ 740.783390] Stack: > [ 740.792577] 812e8dcd 0048 0002 > 882f908c8468 > [ 740.827003] 01bef000 882f928e4600 882f28c23e48 > 882f28c23d70 > [ 740.860971] 882f28c23d38 812e8dcd 0001 > 882f908c8300 > [ 740.894994] Call Trace: > [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 > [ 740.932940] [] fuse_direct_IO+0xdd/0x280 > [ 740.958866] [] generic_file_direct_write+0x9e/0x150 > [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 > [ 741.017725] [] __vfs_write+0xa7/0xf0 > [ 741.041787] [] vfs_write+0xa9/0x190 > [ 741.065307] [] SyS_pwrite64+0x69/0xa0 > [ 741.090141]
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Thank you Seth for taking a close look at the problem and my proposed fix. As mentioned on the mailing list my test runs fine now with the two fixes. However, I prefer your fix as it prevents us from running into this issue again. Our test system is happily installing VMs for two hours now using your build. Please propose your patch. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
The bug triggers with the debug kernel, however there is no message like "fuse_direct_IO: io->reg would have gone negative" in the journal: Jan 29 16:22:18 ubuntu dnsmasq-dhcp[896]: DHCPREQUEST(virbr0) 192.168.122.93 52:54:00:45:1c:61 Jan 29 16:22:18 ubuntu dnsmasq-dhcp[896]: DHCPACK(virbr0) 192.168.122.93 52:54:00:45:1c:61 Jan 29 16:22:51 ubuntu kernel: BUG: unable to handle kernel paging request at 8800904b06c0 Jan 29 16:22:51 ubuntu kernel: IP: [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: PGD 1ff0067 PUD 3738b6063 PMD 0 Jan 29 16:22:51 ubuntu kernel: Oops: [#1] SMP Jan 29 16:22:51 ubuntu kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 ipmi_ssif ipmi_devintf gpio_ich coretemp kvm_intel serio_raw kvm input_leds cdc_ether usbnet mii lpc_ich i7core_edac ioatdma edac_core i5500_temp shpchp dca 8250_fintek ipmi_si mac_hid ipmi_msghandler sunrpc autofs4 hid_generic mptsas mptscsih usbhid mptbase psmouse hid pata_acpi scsi_transport_sas bnx2 Jan 29 16:22:51 ubuntu kernel: CPU: 4 PID: 21954 Comm: qemu-system-x86 Tainted: G I 4.2.0-27-generic #32lp1505948v201601281755 Jan 29 16:22:51 ubuntu kernel: Hardware name: IBM System x3550 M2 -[794654G]-/49Y6512 , BIOS -[D6E131CUS-1.05]- 11/25/2009 Jan 29 16:22:51 ubuntu kernel: task: 880380e98c80 ti: 8803811d4000 task.ti: 8803811d4000 Jan 29 16:22:51 ubuntu kernel: RIP: 0010:[] [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: RSP: 0018:8803811d79c8 EFLAGS: 00010286 Jan 29 16:22:51 ubuntu kernel: RAX: RBX: 00d0 RCX: 0009d36e Jan 29 16:22:51 ubuntu kernel: RDX: 0009d36d RSI: RDI: 00019aa0 Jan 29 16:22:51 ubuntu kernel: RBP: 8803811d7a08 R08: 88067fc19aa0 R09: 812f8d56 Jan 29 16:22:51 ubuntu kernel: R10: 8800904b06c0 R11: 081a R12: 00d0 Jan 29 16:22:51 ubuntu kernel: R13: 0058 R14: 8803738037c0 R15: 8803738037c0 Jan 29 16:22:51 ubuntu kernel: FS: 7f384a78eb00() GS:88067fc0() knlGS: Jan 29 16:22:51 ubuntu kernel: CS: 0010 DS: ES: CR0: 8005003b Jan 29 16:22:51 ubuntu kernel: CR2: 8800904b06c0 CR3: 0002da9d5000 CR4: 26e0 Jan 29 16:22:51 ubuntu kernel: Stack: Jan 29 16:22:51 ubuntu kernel: 8803811d7a18 812f8d56 880371e2b200 8805993ae0d0 Jan 29 16:22:51 ubuntu kernel: 000b 00d0 0058 8805993ae210 Jan 29 16:22:51 ubuntu kernel: 8803811d7a58 812f8d56 8803811d7a38 8805993ae0d0 Jan 29 16:22:51 ubuntu kernel: Call Trace: Jan 29 16:22:51 ubuntu kernel: [] ? __fuse_request_alloc+0x56/0xd0 Jan 29 16:22:51 ubuntu kernel: [] __fuse_request_alloc+0x56/0xd0 Jan 29 16:22:51 ubuntu kernel: [] __fuse_get_req+0x1d6/0x280 Jan 29 16:22:51 ubuntu kernel: [] ? wake_atomic_t_function+0x60/0x60 Jan 29 16:22:51 ubuntu kernel: [] fuse_get_req+0x10/0x20 Jan 29 16:22:51 ubuntu kernel: [] fuse_direct_io+0x4fd/0x5c0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_getxattr+0x12f/0x160 Jan 29 16:22:51 ubuntu kernel: [] ? kmem_cache_alloc_trace+0x187/0x1f0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_direct_IO+0xff/0x3b0 Jan 29 16:22:51 ubuntu kernel: [] fuse_direct_IO+0x193/0x3b0 Jan 29 16:22:51 ubuntu kernel: [] generic_file_direct_write+0xb9/0x180 Jan 29 16:22:51 ubuntu kernel: [] fuse_file_write_iter+0x15c/0x2e0 Jan 29 16:22:51 ubuntu kernel: [] ? security_file_permission+0x3d/0xc0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_perform_write+0x540/0x540 Jan 29 16:22:51 ubuntu kernel: [] aio_run_iocb+0x27f/0x2e0 Jan 29 16:22:51 ubuntu kernel: [] ? fsnotify+0x316/0x4a0 Jan 29 16:22:51 ubuntu kernel: [] ? __fget_light+0x25/0x60 Jan 29 16:22:51 ubuntu kernel: [] do_io_submit+0x24b/0x4f0 Jan 29 16:22:51 ubuntu kernel: [] ? wake_up_q+0x70/0x70 Jan 29 16:22:51 ubuntu kernel: [] SyS_io_submit+0x10/0x20 Jan 29 16:22:51 ubuntu kernel: [] entry_SYSCALL_64_fastpath+0x16/0x75 Jan 29 16:22:51 ubuntu kernel: Code: 08 65 4c 03 05 36 af e2 7e 49 83 78 10 00 4d 8b 10 0f 84 36 01 00 00 4d 85 d2 0f 84 2d 01 00 00 49 63 46 20 48 8d 4a 01 49 8b 3e <49> 8b 1c 02 4c 89 d0 65 48 0f c7 0f 0f 94 c0 84 c0 74 bb 49 63 Jan 29 16:22:51 ubuntu kernel: RIP [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: RSP Jan 29 16:22:51 ubuntu kernel: CR2: 8800904b06c0 Jan 29 16:22:51 ubuntu kernel: ---[ end trace 1ebba465731d9933 ]--- Jan 29 16:22:52 ubuntu kernel: BUG: unable to handle kernel paging request at 8800904b06c0 Jan 29 16:22:52 ubuntu kernel: IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 Jan 29 16:22:52 ubuntu kernel: PGD 1ff0067 PUD 3738b6063 PMD 0 Jan 29
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Enabling KASAN on a Wily kernel prints the following: Jan 27 12:02:05 ubuntu kernel: == Jan 27 12:02:05 ubuntu kernel: BUG: KASan: use after free in fuse_direct_IO+0xb1a/0xcc0 at addr 88036c414390 Jan 27 12:02:05 ubuntu kernel: Read of size 8 by task qemu-system-x86/2784 Jan 27 12:02:05 ubuntu kernel: = Jan 27 12:02:05 ubuntu kernel: BUG kmalloc-128 (Tainted: G I ): kasan: bad access detected Jan 27 12:02:05 ubuntu kernel: - Jan 27 12:02:05 ubuntu kernel: Disabling lock debugging due to kernel taint Jan 27 12:02:05 ubuntu kernel: INFO: Slab 0xea000db10500 objects=32 used=26 fp=0x88036c414e80 flags=0x280 Jan 27 12:02:05 ubuntu kernel: INFO: Object 0x88036c414380 @offset=896 fp=0x (null) Jan 27 12:02:05 ubuntu kernel: Bytes b4 88036c414370: 18 00 00 00 40 27 a3 1f 3b 56 00 00 00 00 00 00 @'..;V.. Jan 27 12:02:05 ubuntu kernel: Object 88036c414380: 00 00 00 00 00 00 00 00 00 f0 75 35 00 00 00 00 ..u5 Jan 27 12:02:05 ubuntu kernel: Object 88036c414390: 80 27 67 81 ff ff ff ff 00 00 00 00 00 00 00 00 .'g. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143a0: 05 00 00 00 00 00 00 00 80 82 44 ad 05 88 ff ff ..D. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143b0: 00 00 00 00 00 00 00 00 10 e1 bc 56 49 56 00 00 ...VIV.. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: Object 88036c4143d0: 00 00 00 00 00 00 00 00 80 f6 85 6d 03 88 ff ff ...m Jan 27 12:02:05 ubuntu kernel: Object 88036c4143e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: Object 88036c4143f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: CPU: 0 PID: 2784 Comm: qemu-system-x86 Tainted: G B I 4.2.0-25-generic 030 Jan 27 12:02:05 ubuntu kernel: Hardware name: IBM System x3550 M2 -[794654G]-/49Y6512 , BIOS -[D6E131CUS-1.05]- 11/25/2009 Jan 27 12:02:05 ubuntu kernel: 88036c414380 d939cde9 8805adf0f7c8 828cafee Jan 27 12:02:05 ubuntu kernel: 0080 880373803680 8805adf0f7f8 81546759 Jan 27 12:02:05 ubuntu kernel: 880373803680 ea000db10500 88036c414380 8805ad56d600 Jan 27 12:02:05 ubuntu kernel: Call Trace: Jan 27 12:02:05 ubuntu kernel: [< inline >] __dump_stack linux-4.2.0/lib/dump_stack.c:15 Jan 27 12:02:05 ubuntu kernel: [] dump_stack+0x45/0x57 linux-4.2.0/lib/dump_stack.c:50 Jan 27 12:02:05 ubuntu kernel: [] print_trailer+0xf9/0x150 linux-4.2.0/mm/slub.c:650 Jan 27 12:02:05 ubuntu kernel: [] object_err+0x38/0x50 linux-4.2.0/mm/slub.c:657 Jan 27 12:02:05 ubuntu kernel: [< inline >] print_address_description linux-4.2.0/mm/kasan/report.c:120 Jan 27 12:02:05 ubuntu kernel: [] kasan_report_error+0x1e8/0x3f0 linux-4.2.0/mm/kasan/report.c:193 Jan 27 12:02:05 ubuntu kernel: [< inline >] kasan_report linux-4.2.0/mm/kasan/report.c:230 Jan 27 12:02:05 ubuntu kernel: [] __asan_report_load8_noabort+0x61/0x70 linux-4.2.0/mm/kasan/report.c:251 Jan 27 12:02:05 ubuntu kernel: [] fuse_direct_IO+0xb1a/0xcc0 linux-4.2.0/fs/fuse/file.c:2842 Jan 27 12:02:05 ubuntu kernel: [] generic_file_direct_write+0x246/0x540 linux-4.2.0/mm/filemap.c:2398 Jan 27 12:02:05 ubuntu kernel: [] fuse_file_write_iter+0x31c/0x780 linux-4.2.0/fs/fuse/file.c:1182 Jan 27 12:02:05 ubuntu kernel: [] aio_run_iocb+0x68a/0x870 linux-4.2.0/fs/aio.c:1446 Jan 27 12:02:05 ubuntu kernel: [< inline >] io_submit_one linux-4.2.0/fs/aio.c:1548 Jan 27 12:02:05 ubuntu kernel: [] do_io_submit+0x4a7/0xb40 linux-4.2.0/fs/aio.c:1606 Jan 27 12:02:05 ubuntu kernel: [< inline >] SYSC_io_submit linux-4.2.0/fs/aio.c:1631 Jan 27 12:02:05 ubuntu kernel: [] SyS_io_submit+0x10/0x20 linux-4.2.0/fs/aio.c:1628 Jan 27 12:02:05 ubuntu kernel: [] entry_SYSCALL_64_fastpath+0x16/0x75 linux-4.2.0/arch/x86/entry/entry_64.S:186 Jan 27 12:02:05 ubuntu kernel: Memory state around the buggy address: Jan 27 12:02:05 ubuntu kernel: 88036c414280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb Jan 27 12:02:05 ubuntu kernel: 88036c414300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: >88036c414380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb Jan 27 12:02:05 ubuntu kernel: ^ Jan 27 12:02:05 ubuntu kernel: 88036c414400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: 88036c414480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc Jan 27 12:02:05 ubuntu kernel: == -- You received this bug notification because you are a member of Kernel Packages, which
[Kernel-packages] [Bug 1505948] Re: Memory allocation failure crashes kernel hard, presumably related to FUSE
Duplicating my post to the fuse developer mailing list here: Hi all, the kernel crash can be triggered if async direct IO is used which comes with Fuse 3.0_pre0 (i.e. current head). My workload was to install CentOS7 on a newly created qcow2 disk. The kernel (Fedora 21; 4.1.8-100.fc21.x86_64) crashed in 2/2 runs using qemu/kvm atop of ntfs- 3g built against fuse3: 1) Build fuse3 from current head 2) Build ntfs-3g against fuse3 (feel free to use the attached patch. It assumes that pkg-config is able to find fuse3, so install fuse3.pc in a PKG_CONFIG_PATH) 3) ntfs-3g: ./configure --with-fuse=external; make 4) "src/lowntfs-3g --version" should now print 'lowntfs-3g 2015.3.14 external FUSE 30' 5) create and mount an NTFS volume 6) create a VM disk: qemu-img create -f qcow2 disk.qcow2 20G 7) make sure that the VM actually uses async direct io (cache='none' io='native') In my case the kernel crashed around 12 minutes after the VM was started. Regards, Robert ** Patch added: "Patch to build ntfs-3g against fuse3" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+attachment/4505214/+files/ntfs3g-fuse3.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory allocation failure crashes kernel hard, presumably related to FUSE Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20
