[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Launchpad has imported 10 comments from the remote bug at https://bugzilla.redhat.com/show_bug.cgi?id=1254310. If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. On 2015-08-17T16:41:10+00:00 Ian wrote: Description of problem: After upgrading a node from F20 to F21, node crashes accessing glusterfs volume. The remaining F20 nodes have no problem accessing the volume. Aug 16 20:24:25 bagel kernel: [ 1810.077267] [ cut here ] Aug 16 20:24:25 bagel kernel: [ 1810.081945] kernel BUG at mm/slub.c:3413! Aug 16 20:24:25 bagel kernel: [ 1810.085998] invalid opcode: [#1] SMP Aug 16 20:24:25 bagel kernel: [ 1810.090177] Modules linked in: vhost_net vhost m acvtap macvlan ebt_arp ebtable_nat fuse nfsv3 nfs_acl nfs lockd grace sunrpc fsca che ebtable_filter ebtables ip6table_filter ip6_tables softdog scsi_transport_isc si xt_physdev br_netfilter nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_connt rack nf_conntrack vfat fat coretemp kvm_intel kvm bcache iTCO_wdt crct10dif_pclmu l ipmi_devintf crc32_pclmul iTCO_vendor_support gpio_ich igb crc32c_intel ptp pps _core lpc_ich ghash_clmulni_intel i2c_i801 mfd_core ipmi_si dca ipmi_msghandler i 2c_ismt tpm_tis shpchp tpm acpi_cpufreq ast i2c_algo_bit drm_kms_helper ttm drm 8 021q garp mrp tun bridge stp llc bonding Aug 16 20:24:25 bagel kernel: [ 1810.149526] CPU: 1 PID: 4794 Comm: qemu-system-x 86 Not tainted 4.1.4-100.fc21.x86_64 #1 Aug 16 20:24:25 bagel kernel: [ 1810.157603] Hardware name: Supermicro A1SRM-2758 F/A1SRM-2758F, BIOS 1.2 02/16/2015 Aug 16 20:24:25 bagel kernel: [ 1810.165246] task: 88085a1313c0 ti: 8803b 09b4000 task.ti: 8803b09b4000 Aug 16 20:24:25 bagel kernel: [ 1810.172800] RIP: 0010:[] [] kfree+0x152/0x160 Aug 16 20:24:25 bagel kernel: [ 1810.180467] RSP: 0018:8803b09b7c98 EFLAGS: 00010246 Aug 16 20:24:25 bagel kernel: [ 1810.185833] RAX: 0058002c RBX: 88020 08b9960 RCX: dead00200200 Aug 16 20:24:25 bagel kernel: [ 1810.193032] RDX: 77ff8000 RSI: 88085 a1313c0 RDI: 8802008b9960 Aug 16 20:24:25 bagel kernel: [ 1810.200231] RBP: 8803b09b7cb8 R08: 8803b 09b7c80 R09: ea0008022e40 Aug 16 20:24:25 bagel kernel: [ 1810.207431] R10: 2fe4 R11: 0 000 R12: 000149928000 Aug 16 20:24:25 bagel kernel: [ 1810.214629] R13: a02e5c8c R14: 8803b 09b7e50 R15: 8801009b5600 Aug 16 20:24:25 bagel kernel: [ 1810.221829] FS: 7f35609ff700() GS:88087fc4() knlGS: Aug 16 20:24:25 bagel kernel: [ 1810.229992] CS: 0010 DS: ES: CR0: 8005003b Aug 16 20:24:25 bagel kernel: [ 1810.235799] CR2: 7fbf24022a98 CR3: 000100a81000 CR4: 001027e0 Aug 16 20:24:25 bagel kernel: [ 1810.243001] Stack: Aug 16 20:24:25 bagel kernel: [ 1810.245037] 8802008b9960 8802008b9960 000149928000 8803b09b7da8 Aug 16 20:24:25 bagel kernel: [ 1810.252590] 8803b09b7d48 a02e5c8c 4800 8806eea842c0 Aug 16 20:24:25 bagel kernel: [ 1810.260145] 4800 0001f400 00014992c800 Aug 16 20:24:25 bagel kernel: [ 1810.267699] Call Trace: Aug 16 20:24:25 bagel kernel: [ 1810.270189] [] fuse_direct_IO+0x20c/0x340 [fuse] Aug 16 20:24:25 bagel kernel: [ 1810.276525] [] generic_file_read_iter+0x4ca/0x600 Aug 16 20:24:25 bagel kernel: [ 1810.282941] [] fuse_file_read_iter+0x4c/0x70 [fuse] Aug 16 20:24:25 bagel kernel: [ 1810.289531] [] __vfs_read+0xce/0x100 Aug 16 20:24:25 bagel kernel: [ 1810.294810] [] vfs_read+0x8a/0x140 Aug 16 20:24:25 bagel kernel: [ 1810.299910] [] SyS_pread64+0x92/0xc0 Aug 16 20:24:25 bagel kernel: [ 1810.305186] [] system_call_fastpath+0x12/0x71 Aug 16 20:24:25 bagel kernel: [ 1810.311253] Code: 00 4d 8b 49 30 e9 35 ff ff ff 0f 1f 80 00 00 00 00 4c 89 d1 48 89 da 4c 89 ce e8 ca f9 ff ff e9 73 ff ff ff 0f 1f 44 00 00 0f 0b <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 89 Aug 16 20:24:25 bagel kernel: [ 1810.336949] RIP [] kfree+0x152/0x160 Aug 16 20:24:25 bagel kernel: [ 1810.344889] RSP Aug 16 20:24:25 bagel kernel: [ 1810.360802] ---[ end trace 76f7ea1ab5ea1b36 ]--- Version-Release number of selected component (if applicable): kernel-4.1.4-100.fc21.x86_64 glusterfs-fuse-3.5.5-2.fc21.x86_64 How reproducible: Every time I would start a VM whose disk lived on the gluster volume, the crash would happen immediately. The node would become mostly unresponsive and require a hard reset. Steps to Reproduce: 1. glusterfs distributed-replicated volume across 3 F20 nodes. 2. upgrade one node from F20 to F21 3. attempt to run a VM on the new F21 node (accessing a disk image on the gluster volume) Actual results:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
This bug was fixed in the package linux - 4.2.0-36.41 --- linux (4.2.0-36.41) wily; urgency=low [ Kamal Mostafa ] * Release Tracking Bug - LP: #1571667 [ Benjamin Tissoires ] * SAUCE: Input: synaptics - handle spurious release of trackstick buttons, again - LP: #1553811 [ dann frazier ] * Revert "SAUCE: arm64, numa, dt: adding dt based numa support using dt node property arm, associativity" - LP: #1558828 * Revert "SAUCE: Documentation: arm64/arm: dt bindings for numa." - LP: #1558828 * Revert "SAUCE: arm64, numa: adding numa support for arm64 platforms." - LP: #1558828 * Revert "[Config] Enable NUMA on ARM64" - LP: #1558828 [ K. Y. Srinivasan ] * SAUCE: (noup): Drivers: hv: vmbus: Fix a bug in hv_need_to_signal_on_read() - LP: #1556264 [ Kamal Mostafa ] * [debian] BugLink: close LP: bugs only for Launchpad urls * [Config] updateconfigs after v4.2.8-ckt7 [ Upstream Kernel Changes ] * Revert "jffs2: Fix lock acquisition order bug in jffs2_write_begin" - LP: #1561677 * tipc: fix connection abort during subscription cancel - LP: #1561677 * tipc: fix nullptr crash during subscription cancel - LP: #1561677 * s390/mm: four page table levels vs. fork - LP: #1561677 * Input: aiptek - fix crash on detecting device without endpoints - LP: #1561677 * wext: fix message delay/ordering - LP: #1561677 * cfg80211/wext: fix message ordering - LP: #1561677 * mac80211: fix use of uninitialised values in RX aggregation - LP: #1561677 * mac80211: minstrel: Change expected throughput unit back to Kbps - LP: #1561677 * libata: fix HDIO_GET_32BIT ioctl - LP: #1561677 * iwlwifi: mvm: inc pending frames counter also when txing non-sta - LP: #1561677 * [media] adv7604: fix tx 5v detect regression - LP: #1561677 * ahci: add new Intel device IDs - LP: #1561677 * ahci: Order SATA device IDs for codename Lewisburg - LP: #1561677 * Adding Intel Lewisburg device IDs for SATA - LP: #1561677 * ASoC: samsung: Use IRQ safe spin lock calls - LP: #1561677 * mac80211: minstrel_ht: set default tx aggregation timeout to 0 - LP: #1561677 * usb: chipidea: otg: change workqueue ci_otg as freezable - LP: #1561677 * jffs2: Fix page lock / f->sem deadlock - LP: #1561677 * Fix directory hardlinks from deleted directories - LP: #1561677 * iommu/amd: Fix boot warning when device 00:00.0 is not iommu covered - LP: #1561677 * iommu/amd: Apply workaround for ATS write permission check - LP: #1561677 * libata: Align ata_device's id on a cacheline - LP: #1561677 * can: gs_usb: fixed disconnect bug by removing erroneous use of kfree() - LP: #1561677 * fbcon: set a default value to blink interval - LP: #1561677 * KVM: x86: fix root cause for missed hardware breakpoints - LP: #1561677 * arm64: vmemmap: use virtual projection of linear region - LP: #1561677 * vfio: fix ioctl error handling - LP: #1561677 * ALSA: ctl: Fix ioctls for X32 ABI - LP: #1561677 * ALSA: pcm: Fix ioctls for X32 ABI - LP: #1561677 * ALSA: rawmidi: Fix ioctls X32 ABI - LP: #1561677 * ALSA: timer: Fix broken compat timer user status ioctl - LP: #1561677 * ALSA: timer: Fix ioctls for X32 ABI - LP: #1561677 * cifs: fix out-of-bounds access in lease parsing - LP: #1561677 * CIFS: Fix SMB2+ interim response processing for read requests - LP: #1561677 * Fix cifs_uniqueid_to_ino_t() function for s390x - LP: #1561677 * arm/arm64: KVM: Fix ioctl error handling - LP: #1561677 * MIPS: kvm: Fix ioctl error handling. - LP: #1561677 * ALSA: hdspm: Fix wrong boolean ctl value accesses - LP: #1561677 * ALSA: hdspm: Fix zero-division - LP: #1561677 * ALSA: hdsp: Fix wrong boolean ctl value accesses - LP: #1561677 * use ->d_seq to get coherency between ->d_inode and ->d_flags - LP: #1561677 * USB: qcserial: add Dell Wireless 5809e Gobi 4G HSPA+ (rev3) - LP: #1561677 * USB: cp210x: Add ID for Parrot NMEA GPS Flight Recorder - LP: #1561677 * ASoC: dapm: Fix ctl value accesses in a wrong type - LP: #1561677 * ASoC: wm8958: Fix enum ctl accesses in a wrong type - LP: #1561677 * ASoC: wm8994: Fix enum ctl accesses in a wrong type - LP: #1561677 * ASoC: wm_adsp: Fix enum ctl accesses in a wrong type - LP: #1561677 * USB: serial: option: add support for Telit LE922 PID 0x1045 - LP: #1561677 * USB: serial: option: add support for Quectel UC20 - LP: #1561677 * ALSA: usb-audio: Add a quirk for Plantronics DA45 - LP: #1561677 * mac80211: check PN correctly for GCMP-encrypted fragmented MPDUs - LP: #1561677 * mac80211: Fix Public Action frame RX in AP mode - LP: #1561677 * i2c: brcmstb: allocate correct amount of memory for regmap - LP: #1561677 * ALSA: seq: oss: Don't drain at closing a client - LP:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
This bug was fixed in the package linux - 4.2.0-36.41 --- linux (4.2.0-36.41) wily; urgency=low [ Kamal Mostafa ] * Release Tracking Bug - LP: #1571667 [ Benjamin Tissoires ] * SAUCE: Input: synaptics - handle spurious release of trackstick buttons, again - LP: #1553811 [ dann frazier ] * Revert "SAUCE: arm64, numa, dt: adding dt based numa support using dt node property arm, associativity" - LP: #1558828 * Revert "SAUCE: Documentation: arm64/arm: dt bindings for numa." - LP: #1558828 * Revert "SAUCE: arm64, numa: adding numa support for arm64 platforms." - LP: #1558828 * Revert "[Config] Enable NUMA on ARM64" - LP: #1558828 [ K. Y. Srinivasan ] * SAUCE: (noup): Drivers: hv: vmbus: Fix a bug in hv_need_to_signal_on_read() - LP: #1556264 [ Kamal Mostafa ] * [debian] BugLink: close LP: bugs only for Launchpad urls * [Config] updateconfigs after v4.2.8-ckt7 [ Upstream Kernel Changes ] * Revert "jffs2: Fix lock acquisition order bug in jffs2_write_begin" - LP: #1561677 * tipc: fix connection abort during subscription cancel - LP: #1561677 * tipc: fix nullptr crash during subscription cancel - LP: #1561677 * s390/mm: four page table levels vs. fork - LP: #1561677 * Input: aiptek - fix crash on detecting device without endpoints - LP: #1561677 * wext: fix message delay/ordering - LP: #1561677 * cfg80211/wext: fix message ordering - LP: #1561677 * mac80211: fix use of uninitialised values in RX aggregation - LP: #1561677 * mac80211: minstrel: Change expected throughput unit back to Kbps - LP: #1561677 * libata: fix HDIO_GET_32BIT ioctl - LP: #1561677 * iwlwifi: mvm: inc pending frames counter also when txing non-sta - LP: #1561677 * [media] adv7604: fix tx 5v detect regression - LP: #1561677 * ahci: add new Intel device IDs - LP: #1561677 * ahci: Order SATA device IDs for codename Lewisburg - LP: #1561677 * Adding Intel Lewisburg device IDs for SATA - LP: #1561677 * ASoC: samsung: Use IRQ safe spin lock calls - LP: #1561677 * mac80211: minstrel_ht: set default tx aggregation timeout to 0 - LP: #1561677 * usb: chipidea: otg: change workqueue ci_otg as freezable - LP: #1561677 * jffs2: Fix page lock / f->sem deadlock - LP: #1561677 * Fix directory hardlinks from deleted directories - LP: #1561677 * iommu/amd: Fix boot warning when device 00:00.0 is not iommu covered - LP: #1561677 * iommu/amd: Apply workaround for ATS write permission check - LP: #1561677 * libata: Align ata_device's id on a cacheline - LP: #1561677 * can: gs_usb: fixed disconnect bug by removing erroneous use of kfree() - LP: #1561677 * fbcon: set a default value to blink interval - LP: #1561677 * KVM: x86: fix root cause for missed hardware breakpoints - LP: #1561677 * arm64: vmemmap: use virtual projection of linear region - LP: #1561677 * vfio: fix ioctl error handling - LP: #1561677 * ALSA: ctl: Fix ioctls for X32 ABI - LP: #1561677 * ALSA: pcm: Fix ioctls for X32 ABI - LP: #1561677 * ALSA: rawmidi: Fix ioctls X32 ABI - LP: #1561677 * ALSA: timer: Fix broken compat timer user status ioctl - LP: #1561677 * ALSA: timer: Fix ioctls for X32 ABI - LP: #1561677 * cifs: fix out-of-bounds access in lease parsing - LP: #1561677 * CIFS: Fix SMB2+ interim response processing for read requests - LP: #1561677 * Fix cifs_uniqueid_to_ino_t() function for s390x - LP: #1561677 * arm/arm64: KVM: Fix ioctl error handling - LP: #1561677 * MIPS: kvm: Fix ioctl error handling. - LP: #1561677 * ALSA: hdspm: Fix wrong boolean ctl value accesses - LP: #1561677 * ALSA: hdspm: Fix zero-division - LP: #1561677 * ALSA: hdsp: Fix wrong boolean ctl value accesses - LP: #1561677 * use ->d_seq to get coherency between ->d_inode and ->d_flags - LP: #1561677 * USB: qcserial: add Dell Wireless 5809e Gobi 4G HSPA+ (rev3) - LP: #1561677 * USB: cp210x: Add ID for Parrot NMEA GPS Flight Recorder - LP: #1561677 * ASoC: dapm: Fix ctl value accesses in a wrong type - LP: #1561677 * ASoC: wm8958: Fix enum ctl accesses in a wrong type - LP: #1561677 * ASoC: wm8994: Fix enum ctl accesses in a wrong type - LP: #1561677 * ASoC: wm_adsp: Fix enum ctl accesses in a wrong type - LP: #1561677 * USB: serial: option: add support for Telit LE922 PID 0x1045 - LP: #1561677 * USB: serial: option: add support for Quectel UC20 - LP: #1561677 * ALSA: usb-audio: Add a quirk for Plantronics DA45 - LP: #1561677 * mac80211: check PN correctly for GCMP-encrypted fragmented MPDUs - LP: #1561677 * mac80211: Fix Public Action frame RX in AP mode - LP: #1561677 * i2c: brcmstb: allocate correct amount of memory for regmap - LP: #1561677 * ALSA: seq: oss: Don't drain at closing a client - LP:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Done. ** Tags removed: verification-needed-wily ** Tags added: verification-done-wily -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Released Status in linux source package in Wily: Fix Committed Status in linux source package in Xenial: Fix Released Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use- after-free bugs which are causing kernel crashes. Fix: Two commits from fuse-next, one which simply caches the result of a test to avoid a use-after-free and another which adds reference counting to the fuse_io_priv struct to get rid of some convoluted rules for determining when this structure can be freed. Test case: Tested on LP #1505948. --- Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- wily' to 'verification-done-wily'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-wily -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Released Status in linux source package in Wily: Fix Committed Status in linux source package in Xenial: Fix Released Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use- after-free bugs which are causing kernel crashes. Fix: Two commits from fuse-next, one which simply caches the result of a test to avoid a use-after-free and another which adds reference counting to the fuse_io_priv struct to get rid of some convoluted rules for determining when this structure can be freed. Test case: Tested on LP #1505948. --- Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Branch linked: lp:ubuntu/trusty-proposed/linux-lts-wily -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Released Status in linux source package in Wily: Fix Committed Status in linux source package in Xenial: Fix Released Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use- after-free bugs which are causing kernel crashes. Fix: Two commits from fuse-next, one which simply caches the result of a test to avoid a use-after-free and another which adds reference counting to the fuse_io_priv struct to get rid of some convoluted rules for determining when this structure can be freed. Test case: Tested on LP #1505948. --- Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Changed in: linux (Ubuntu Wily) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Released Status in linux source package in Wily: Fix Committed Status in linux source package in Xenial: Fix Released Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use- after-free bugs which are causing kernel crashes. Fix: Two commits from fuse-next, one which simply caches the result of a test to avoid a use-after-free and another which adds reference counting to the fuse_io_priv struct to get rid of some convoluted rules for determining when this structure can be freed. Test case: Tested on LP #1505948. --- Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
This bug was fixed in the package linux - 4.4.0-16.32 --- linux (4.4.0-16.32) xenial; urgency=low [ Tim Gardner ] * Release Tracking Bug - LP: #1561727 * fix thermal throttling due to commit "Thermal: initialize thermal zone device correctly" (LP: #1561676) - Thermal: Ignore invalid trip points * Thinkpad T460: Trackpoint mouse buttons instantly generate "release" event on press (LP: #1553811) - SAUCE: (noup) Input: synaptics - handle spurious release of trackstick buttons, again * reading /sys/kernel/security/apparmor/profiles requires CAP_MAC_ADMIN (LP: #1560583) - SAUCE: apparmor: Allow ns_root processes to open profiles file - SAUCE: apparmor: Consult sysctl when reading profiles in a user ns * linux: sync virtualbox drivers to 5.0.16-dfsg-2 (LP: #1561492) - ubuntu: vbox -- update to 5.0.16-dfsg-2 * s390/kconfig: CONFIG_NUMA without CONFIG_NUMA_EMU does not make any sense on s390x (LP: #1557690) - [Config] CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=n for s390x * spl/zfs fails to build on s390x (LP: #1519814) - [Config] s390x -- re-enable zfs - [Config] zfs -- disable powerpc until the test failures can be resolved * linux: sync to ZFS 0.6.5.6 stable release (LP: #1561483) - SAUCE: (noup) Update spl to 0.6.5.6-0ubuntu1, zfs to 0.6.5.6-0ubuntu1 * zfs: enable zfs for 64bit powerpc kernels (LP: #1558871) - [Packaging] zfs -- handle rprovides via dpkg-gencontrol - [Config] powerpc -- convert zfs configuration to custom_override * Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) (LP: #1505948) - SAUCE: (noup) fuse: do not use iocb after it may have been freed - SAUCE: (noup) fuse: Add reference counting for fuse_io_priv * cgroup namespaces: add a 'nsroot=' mountinfo field (LP: #1560489) - SAUCE: (noup) cgroup namespaces: add a 'nsroot=' mountinfo field * linux packaging: clear remaining redundant delta (LP: #1560445) - [Debian] Remove generated intermediate files on clean * arm64: guest hangs when ntpd is running (LP: #1549494) - Revert "hrtimer: Add support for CLOCK_MONOTONIC_RAW" - Revert "hrtimer: Catch illegal clockids" - Revert "KVM: arm/arm64: timer: Switch to CLOCK_MONOTONIC_RAW" * Need enough contiguous memory to support GICv3 ITS table (LP: #1558828) - [Config] CONFIG_FORCE_MAX_ZONEORDER=13 on arm64 - SAUCE: (no-up) arm64: gicv3: its: Increase FORCE_MAX_ZONEORDER for Cavium ThunderX * update arcmsr to version v1.30.00.22-20151126 to fix card timeouts (LP: #1559609) - arcmsr: fixed getting wrong configuration data - arcmsr: fixes not release allocated resource - arcmsr: make code more readable - arcmsr: adds code to support new Areca adapter ARC1203 - arcmsr: changes driver version number - arcmsr: more readability improvements - arcmsr: Split dma resource allocation to a new function - arcmsr: change driver version to v1.30.00.22-20151126 * server image has no keyboard, desktop image works (LP: #1559692) - [Config] Rework input-modules (d-i) list * PMU support for Cavium ThunderX (LP: #1559349) - arm64: perf: Rename Cortex A57 events - arm64/perf: Add Cavium ThunderX PMU support - arm64: perf: Enable PMCR long cycle counter bit - arm64: perf: Extend event mask for ARMv8.1 - arm64: dts: Add Cavium ThunderX specific PMU * Show ARM PMU events in perf stat (LP: #1559350) - drivers/perf: kill armpmu_register - arm: perf: Convert event enums to #defines - arm: perf: Add event descriptions - arm64: perf: Convert event enums to #defines - arm64: perf: Add event descriptions - ARM: perf: add format entry to describe event -> config mapping - arm64: perf: add format entry to describe event -> config mapping * [Bug]HSW/BDW EDAC driver reports wrong DIMM (LP: #1559904) - EDAC/sb_edac: Fix computation of channel address * 5-10 second delay in kernel boot with kernel command line ip= (LP: #1259861) - [Config] disable CONFIG_IP_PNP * Miscellaneous Ubuntu changes - [Debian] Silence the reconstruct script -- Tim GardnerMon, 21 Mar 2016 10:15:31 -0600 ** Changed in: linux (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Released Status in linux source package in Wily: In Progress Status in linux source package in Xenial: Fix Released Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use-
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Changed in: linux (Ubuntu Xenial) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Fix Committed Status in linux source package in Wily: In Progress Status in linux source package in Xenial: Fix Committed Status in linux package in Fedora: Unknown Bug description: == SRU Justification == Impact: Races in fuse's synchronous io handling can result in use- after-free bugs which are causing kernel crashes. Fix: Two commits from fuse-next, one which simply caches the result of a test to avoid a use-after-free and another which adds reference counting to the fuse_io_priv struct to get rid of some convoluted rules for determining when this structure can be freed. Test case: Tested on LP #1505948. --- Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Description changed: + == SRU Justification == + + Impact: Races in fuse's synchronous io handling can result in use-after- + free bugs which are causing kernel crashes. + + Fix: Two commits from fuse-next, one which simply caches the result of a + test to avoid a use-after-free and another which adds reference counting + to the fuse_io_priv struct to get rid of some convoluted rules for + determining when this structure can be freed. + + Test case: Tested on LP #1505948. + + --- + Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. ** Changed in: linux (Ubuntu Wily) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu Wily) Assignee: (unassigned) => Seth Forshee (sforshee) ** Changed in: linux (Ubuntu Xenial) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu Xenial) Assignee: (unassigned) => Seth Forshee
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Also affects: linux (Ubuntu Xenial) Importance: High Status: Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux source package in Xenial: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe :
Re: [Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Great, thanks! Robert Am 11.03.2016 15:01 schrieb "Seth Forshee": > On Fri, Mar 11, 2016 at 01:03:32PM -, Robert Doebbelin wrote: > > Thank you Seth for taking a close look at the problem and my proposed > > fix. As mentioned on the mailing list my test runs fine now with the two > > fixes. > > > > However, I prefer your fix as it prevents us from running into this > > issue again. Our test system is happily installing VMs for two hours now > > using your build. Please propose your patch. > > I'm not subscribed to fuse-devel and hadn't refreshed the mailing list > thread so I didn't realize that you had discovered that the hang was > unrelated. That's good. > > I'm happy to send the patches, I'll go ahead and send both my patch and > your iocb patch after I make sure it all applies/builds okay on 4.5. > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1505948 > > Title: > Memory arena corruption with FUSE (was Memory allocation failure > crashes kernel hard, presumably related to FUSE) > > Status in linux package in Ubuntu: > Confirmed > Status in linux source package in Wily: > Confirmed > Status in linux package in Fedora: > Unknown > > Bug description: > Hello everybody, > > Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our > setup when trying to start a Qemu process on top of a fuse-based > mount. Here is an example stacktrace: > > [ 739.807817] BUG: unable to handle kernel paging request at > 8800a4104ea0 > [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 > [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 > [ 739.890418] Oops: [#1] SMP > [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre > ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa > ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi > ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter > xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit > nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack > ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi > x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul > crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul > glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost > macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac > 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek > ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 > sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure > hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core > hpsa mdio wmi > [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted > 4.2.0-040200-generic #201508301530 > [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 > [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: > 882f28c2 > [ 740.451672] RIP: 0010:[] [] > kmem_cache_alloc_trace+0x7a/0x1f0 > [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 > [ 740.518425] RAX: RBX: 00d0 RCX: > 26b3 > [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: > 882fbf407840 > [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: > e8d000200ab0 > [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: > 00d0 > [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: > 882fbf407840 > [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() > knlGS: > [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 > [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: > 001426e0 > [ 740.783390] Stack: > [ 740.792577] 812e8dcd 0048 0002 > 882f908c8468 > [ 740.827003] 01bef000 882f928e4600 882f28c23e48 > 882f28c23d70 > [ 740.860971] 882f28c23d38 812e8dcd 0001 > 882f908c8300 > [ 740.894994] Call Trace: > [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 > [ 740.932940] [] fuse_direct_IO+0xdd/0x280 > [ 740.958866] [] generic_file_direct_write+0x9e/0x150 > [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 > [ 741.017725] [] __vfs_write+0xa7/0xf0 > [ 741.041787] [] vfs_write+0xa9/0x190 > [ 741.065307] [] SyS_pwrite64+0x69/0xa0 > [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 > [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 > [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 > 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b > 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 > [ 741.306817]
Re: [Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
On Fri, Mar 11, 2016 at 01:03:32PM -, Robert Doebbelin wrote: > Thank you Seth for taking a close look at the problem and my proposed > fix. As mentioned on the mailing list my test runs fine now with the two > fixes. > > However, I prefer your fix as it prevents us from running into this > issue again. Our test system is happily installing VMs for two hours now > using your build. Please propose your patch. I'm not subscribed to fuse-devel and hadn't refreshed the mailing list thread so I didn't realize that you had discovered that the hang was unrelated. That's good. I'm happy to send the patches, I'll go ahead and send both my patch and your iocb patch after I make sure it all applies/builds okay on 4.5. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Thank you Seth for taking a close look at the problem and my proposed fix. As mentioned on the mailing list my test runs fine now with the two fixes. However, I prefer your fix as it prevents us from running into this issue again. Our test system is happily installing VMs for two hours now using your build. Please propose your patch. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
I don't seem to be able to reproduce. I did try making a patch though that you can try that adds a separate reference count to fuse_io_priv separate from the request count. I don't know if it fixes anything that moving spin_unlock() doesn't, but to me this seems more straightforward and less error prone than having the request count serve kind of as a reference count but not really. A build with my patch and the iocb use-after-free fix are at http://people.canonical.com/~sforshee/lp1505948/. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
I've been looking at the code, but I haven't found anything aside from the two races mentioned on the mailing list thread. Those could explain the original problems, but I don't have any ideas about the problems seen with the fixes applied yet. I'm trying to reproduce now using the steps you provided in xenial but am not having any luck. My vm installed just fine and has been running for half an hour now with some synthesized disk IO. Anything you might have forgot to mention in the steps - ntfs-3g mount options, sepcific version of ntfs-3g to use, etc? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Bug watch added: Red Hat Bugzilla #1254310 https://bugzilla.redhat.com/show_bug.cgi?id=1254310 ** Also affects: linux (Fedora) via https://bugzilla.redhat.com/show_bug.cgi?id=1254310 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Status in linux package in Fedora: Unknown Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions -- Mailing list:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
This also affects the Xenial Standard Kernel. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
The bug triggers with the debug kernel, however there is no message like "fuse_direct_IO: io->reg would have gone negative" in the journal: Jan 29 16:22:18 ubuntu dnsmasq-dhcp[896]: DHCPREQUEST(virbr0) 192.168.122.93 52:54:00:45:1c:61 Jan 29 16:22:18 ubuntu dnsmasq-dhcp[896]: DHCPACK(virbr0) 192.168.122.93 52:54:00:45:1c:61 Jan 29 16:22:51 ubuntu kernel: BUG: unable to handle kernel paging request at 8800904b06c0 Jan 29 16:22:51 ubuntu kernel: IP: [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: PGD 1ff0067 PUD 3738b6063 PMD 0 Jan 29 16:22:51 ubuntu kernel: Oops: [#1] SMP Jan 29 16:22:51 ubuntu kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 ipmi_ssif ipmi_devintf gpio_ich coretemp kvm_intel serio_raw kvm input_leds cdc_ether usbnet mii lpc_ich i7core_edac ioatdma edac_core i5500_temp shpchp dca 8250_fintek ipmi_si mac_hid ipmi_msghandler sunrpc autofs4 hid_generic mptsas mptscsih usbhid mptbase psmouse hid pata_acpi scsi_transport_sas bnx2 Jan 29 16:22:51 ubuntu kernel: CPU: 4 PID: 21954 Comm: qemu-system-x86 Tainted: G I 4.2.0-27-generic #32lp1505948v201601281755 Jan 29 16:22:51 ubuntu kernel: Hardware name: IBM System x3550 M2 -[794654G]-/49Y6512 , BIOS -[D6E131CUS-1.05]- 11/25/2009 Jan 29 16:22:51 ubuntu kernel: task: 880380e98c80 ti: 8803811d4000 task.ti: 8803811d4000 Jan 29 16:22:51 ubuntu kernel: RIP: 0010:[] [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: RSP: 0018:8803811d79c8 EFLAGS: 00010286 Jan 29 16:22:51 ubuntu kernel: RAX: RBX: 00d0 RCX: 0009d36e Jan 29 16:22:51 ubuntu kernel: RDX: 0009d36d RSI: RDI: 00019aa0 Jan 29 16:22:51 ubuntu kernel: RBP: 8803811d7a08 R08: 88067fc19aa0 R09: 812f8d56 Jan 29 16:22:51 ubuntu kernel: R10: 8800904b06c0 R11: 081a R12: 00d0 Jan 29 16:22:51 ubuntu kernel: R13: 0058 R14: 8803738037c0 R15: 8803738037c0 Jan 29 16:22:51 ubuntu kernel: FS: 7f384a78eb00() GS:88067fc0() knlGS: Jan 29 16:22:51 ubuntu kernel: CS: 0010 DS: ES: CR0: 8005003b Jan 29 16:22:51 ubuntu kernel: CR2: 8800904b06c0 CR3: 0002da9d5000 CR4: 26e0 Jan 29 16:22:51 ubuntu kernel: Stack: Jan 29 16:22:51 ubuntu kernel: 8803811d7a18 812f8d56 880371e2b200 8805993ae0d0 Jan 29 16:22:51 ubuntu kernel: 000b 00d0 0058 8805993ae210 Jan 29 16:22:51 ubuntu kernel: 8803811d7a58 812f8d56 8803811d7a38 8805993ae0d0 Jan 29 16:22:51 ubuntu kernel: Call Trace: Jan 29 16:22:51 ubuntu kernel: [] ? __fuse_request_alloc+0x56/0xd0 Jan 29 16:22:51 ubuntu kernel: [] __fuse_request_alloc+0x56/0xd0 Jan 29 16:22:51 ubuntu kernel: [] __fuse_get_req+0x1d6/0x280 Jan 29 16:22:51 ubuntu kernel: [] ? wake_atomic_t_function+0x60/0x60 Jan 29 16:22:51 ubuntu kernel: [] fuse_get_req+0x10/0x20 Jan 29 16:22:51 ubuntu kernel: [] fuse_direct_io+0x4fd/0x5c0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_getxattr+0x12f/0x160 Jan 29 16:22:51 ubuntu kernel: [] ? kmem_cache_alloc_trace+0x187/0x1f0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_direct_IO+0xff/0x3b0 Jan 29 16:22:51 ubuntu kernel: [] fuse_direct_IO+0x193/0x3b0 Jan 29 16:22:51 ubuntu kernel: [] generic_file_direct_write+0xb9/0x180 Jan 29 16:22:51 ubuntu kernel: [] fuse_file_write_iter+0x15c/0x2e0 Jan 29 16:22:51 ubuntu kernel: [] ? security_file_permission+0x3d/0xc0 Jan 29 16:22:51 ubuntu kernel: [] ? fuse_perform_write+0x540/0x540 Jan 29 16:22:51 ubuntu kernel: [] aio_run_iocb+0x27f/0x2e0 Jan 29 16:22:51 ubuntu kernel: [] ? fsnotify+0x316/0x4a0 Jan 29 16:22:51 ubuntu kernel: [] ? __fget_light+0x25/0x60 Jan 29 16:22:51 ubuntu kernel: [] do_io_submit+0x24b/0x4f0 Jan 29 16:22:51 ubuntu kernel: [] ? wake_up_q+0x70/0x70 Jan 29 16:22:51 ubuntu kernel: [] SyS_io_submit+0x10/0x20 Jan 29 16:22:51 ubuntu kernel: [] entry_SYSCALL_64_fastpath+0x16/0x75 Jan 29 16:22:51 ubuntu kernel: Code: 08 65 4c 03 05 36 af e2 7e 49 83 78 10 00 4d 8b 10 0f 84 36 01 00 00 4d 85 d2 0f 84 2d 01 00 00 49 63 46 20 48 8d 4a 01 49 8b 3e <49> 8b 1c 02 4c 89 d0 65 48 0f c7 0f 0f 94 c0 84 c0 74 bb 49 63 Jan 29 16:22:51 ubuntu kernel: RIP [] __kmalloc+0x94/0x250 Jan 29 16:22:51 ubuntu kernel: RSP Jan 29 16:22:51 ubuntu kernel: CR2: 8800904b06c0 Jan 29 16:22:51 ubuntu kernel: ---[ end trace 1ebba465731d9933 ]--- Jan 29 16:22:52 ubuntu kernel: BUG: unable to handle kernel paging request at 8800904b06c0 Jan 29 16:22:52 ubuntu kernel: IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 Jan 29 16:22:52 ubuntu kernel: PGD 1ff0067 PUD 3738b6063 PMD 0 Jan 29
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Interesting that implies that we submitted some kind of async IO, and the IO must have completed and free(io). This implies that the io->req count is getting out of sync with the world. A quick eyeball says we are handling them right, but something is exploding. To try and confirm this is correct I have built a test kernel with a debugging patch applied. This bumps the io->req from 1 (the pending report for the submission of the IO) to 100. If the theory is right the io->req should go to 99 or fewer. If that occurs we should be able to detect it and report the type of the IO in flight. I also have tried to correct for it in the case where that is possible. Would you be able to test the kernel at the below URL and let me know what you see in dmesg. If the detection triggers we should see "fuse_direct_IO: io->reg would have gone negative" messages, and I would be interested in the content of those when it occurs: http://people.canonical.com/~apw/lp1505948-wily/ Builds will be there shortly. Please report any results back here. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
We've been able to confirm an out of bounds write in fuse_direct_io with the slub_debug boot option on linux-lts-wily. ** Attachment added: "Screen Shot 2016-01-26 at 10.00.03.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+attachment/4557543/+files/Screen%20Shot%202016-01-26%20at%2010.00.03.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
We've been able to confirm an out of bounds write in fuse_direct_io with the slub_debug boot option on linux-lts-wily. ** Attachment added: "Screen Shot 2016-01-26 at 10.00.03.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+attachment/4557544/+files/Screen%20Shot%202016-01-26%20at%2010.00.03.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to:
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
Enabling KASAN on a Wily kernel prints the following: Jan 27 12:02:05 ubuntu kernel: == Jan 27 12:02:05 ubuntu kernel: BUG: KASan: use after free in fuse_direct_IO+0xb1a/0xcc0 at addr 88036c414390 Jan 27 12:02:05 ubuntu kernel: Read of size 8 by task qemu-system-x86/2784 Jan 27 12:02:05 ubuntu kernel: = Jan 27 12:02:05 ubuntu kernel: BUG kmalloc-128 (Tainted: G I ): kasan: bad access detected Jan 27 12:02:05 ubuntu kernel: - Jan 27 12:02:05 ubuntu kernel: Disabling lock debugging due to kernel taint Jan 27 12:02:05 ubuntu kernel: INFO: Slab 0xea000db10500 objects=32 used=26 fp=0x88036c414e80 flags=0x280 Jan 27 12:02:05 ubuntu kernel: INFO: Object 0x88036c414380 @offset=896 fp=0x (null) Jan 27 12:02:05 ubuntu kernel: Bytes b4 88036c414370: 18 00 00 00 40 27 a3 1f 3b 56 00 00 00 00 00 00 @'..;V.. Jan 27 12:02:05 ubuntu kernel: Object 88036c414380: 00 00 00 00 00 00 00 00 00 f0 75 35 00 00 00 00 ..u5 Jan 27 12:02:05 ubuntu kernel: Object 88036c414390: 80 27 67 81 ff ff ff ff 00 00 00 00 00 00 00 00 .'g. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143a0: 05 00 00 00 00 00 00 00 80 82 44 ad 05 88 ff ff ..D. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143b0: 00 00 00 00 00 00 00 00 10 e1 bc 56 49 56 00 00 ...VIV.. Jan 27 12:02:05 ubuntu kernel: Object 88036c4143c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: Object 88036c4143d0: 00 00 00 00 00 00 00 00 80 f6 85 6d 03 88 ff ff ...m Jan 27 12:02:05 ubuntu kernel: Object 88036c4143e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: Object 88036c4143f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: CPU: 0 PID: 2784 Comm: qemu-system-x86 Tainted: G B I 4.2.0-25-generic 030 Jan 27 12:02:05 ubuntu kernel: Hardware name: IBM System x3550 M2 -[794654G]-/49Y6512 , BIOS -[D6E131CUS-1.05]- 11/25/2009 Jan 27 12:02:05 ubuntu kernel: 88036c414380 d939cde9 8805adf0f7c8 828cafee Jan 27 12:02:05 ubuntu kernel: 0080 880373803680 8805adf0f7f8 81546759 Jan 27 12:02:05 ubuntu kernel: 880373803680 ea000db10500 88036c414380 8805ad56d600 Jan 27 12:02:05 ubuntu kernel: Call Trace: Jan 27 12:02:05 ubuntu kernel: [< inline >] __dump_stack linux-4.2.0/lib/dump_stack.c:15 Jan 27 12:02:05 ubuntu kernel: [] dump_stack+0x45/0x57 linux-4.2.0/lib/dump_stack.c:50 Jan 27 12:02:05 ubuntu kernel: [] print_trailer+0xf9/0x150 linux-4.2.0/mm/slub.c:650 Jan 27 12:02:05 ubuntu kernel: [] object_err+0x38/0x50 linux-4.2.0/mm/slub.c:657 Jan 27 12:02:05 ubuntu kernel: [< inline >] print_address_description linux-4.2.0/mm/kasan/report.c:120 Jan 27 12:02:05 ubuntu kernel: [] kasan_report_error+0x1e8/0x3f0 linux-4.2.0/mm/kasan/report.c:193 Jan 27 12:02:05 ubuntu kernel: [< inline >] kasan_report linux-4.2.0/mm/kasan/report.c:230 Jan 27 12:02:05 ubuntu kernel: [] __asan_report_load8_noabort+0x61/0x70 linux-4.2.0/mm/kasan/report.c:251 Jan 27 12:02:05 ubuntu kernel: [] fuse_direct_IO+0xb1a/0xcc0 linux-4.2.0/fs/fuse/file.c:2842 Jan 27 12:02:05 ubuntu kernel: [] generic_file_direct_write+0x246/0x540 linux-4.2.0/mm/filemap.c:2398 Jan 27 12:02:05 ubuntu kernel: [] fuse_file_write_iter+0x31c/0x780 linux-4.2.0/fs/fuse/file.c:1182 Jan 27 12:02:05 ubuntu kernel: [] aio_run_iocb+0x68a/0x870 linux-4.2.0/fs/aio.c:1446 Jan 27 12:02:05 ubuntu kernel: [< inline >] io_submit_one linux-4.2.0/fs/aio.c:1548 Jan 27 12:02:05 ubuntu kernel: [] do_io_submit+0x4a7/0xb40 linux-4.2.0/fs/aio.c:1606 Jan 27 12:02:05 ubuntu kernel: [< inline >] SYSC_io_submit linux-4.2.0/fs/aio.c:1631 Jan 27 12:02:05 ubuntu kernel: [] SyS_io_submit+0x10/0x20 linux-4.2.0/fs/aio.c:1628 Jan 27 12:02:05 ubuntu kernel: [] entry_SYSCALL_64_fastpath+0x16/0x75 linux-4.2.0/arch/x86/entry/entry_64.S:186 Jan 27 12:02:05 ubuntu kernel: Memory state around the buggy address: Jan 27 12:02:05 ubuntu kernel: 88036c414280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb Jan 27 12:02:05 ubuntu kernel: 88036c414300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: >88036c414380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb Jan 27 12:02:05 ubuntu kernel: ^ Jan 27 12:02:05 ubuntu kernel: 88036c414400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jan 27 12:02:05 ubuntu kernel: 88036c414480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc Jan 27 12:02:05 ubuntu kernel: == -- You received this bug notification because you are a member of Kernel Packages, which
[Kernel-packages] [Bug 1505948] Re: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE)
** Summary changed: - Memory allocation failure crashes kernel hard, presumably related to FUSE + Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505948 Title: Memory arena corruption with FUSE (was Memory allocation failure crashes kernel hard, presumably related to FUSE) Status in linux package in Ubuntu: Confirmed Status in linux source package in Wily: Confirmed Bug description: Hello everybody, Linux 4.1, 4.2 or 4.3-rc leads to an immediate kernel panic in our setup when trying to start a Qemu process on top of a fuse-based mount. Here is an example stacktrace: [ 739.807817] BUG: unable to handle kernel paging request at 8800a4104ea0 [ 739.840201] IP: [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 739.870309] PGD 2fee067 PUD 2fbf4dd063 PMD 0 [ 739.890418] Oops: [#1] SMP [ 739.905265] Modules linked in: nbd vport_vxlan vport_gre gre ebtable_filter ebtables openvswitch ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_CT iptable_raw ip_tables xt_tcpudp ip6t_REJECT nf_reject_ipv6 xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables dm_crypt ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd kvm_intel kvm ipmi_devintf vhost_net vhost macvtap macvlan joydev input_leds dm_multipath scsi_dh bonding sb_edac 8021q garp hpilo mrp stp ipmi_si llc edac_core lpc_ich ioatdma 8250_fintek ipmi_msghandler lp shpchp acpi_power_meter mac_hid parport nls_iso8859_1 sch_fq_codel xfs libcrc32c btrfs xor raid6_pq ixgbe ses enclosure hid_generic dca vxlan usbhid ip6_udp_tunnel tg3 udp_tunnel ptp hid pps_core hpsa mdio wmi [ 740.345300] CPU: 8 PID: 10550 Comm: qemu-system-x86 Not tainted 4.2.0-040200-generic #201508301530 [ 740.386879] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015 [ 740.416827] task: 882f8e958dc0 ti: 882f28c2 task.ti: 882f28c2 [ 740.451672] RIP: 0010:[] [] kmem_cache_alloc_trace+0x7a/0x1f0 [ 740.494047] RSP: 0018:882f28c23c68 EFLAGS: 00010286 [ 740.518425] RAX: RBX: 00d0 RCX: 26b3 [ 740.551611] RDX: 26b2 RSI: 00d0 RDI: 882fbf407840 [ 740.584846] RBP: 882f28c23ca8 R08: 00019920 R09: e8d000200ab0 [ 740.618287] R10: 812e8dcd R11: ea00bca0ac00 R12: 00d0 [ 740.651320] R13: 882fbf407840 R14: 8800a4104ea0 R15: 882fbf407840 [ 740.684195] FS: 7f2642ffd700() GS:882fbfa0() knlGS: [ 740.722030] CS: 0010 DS: ES: CR0: 80050033 [ 740.749469] CR2: 8800a4104ea0 CR3: 002f26f83000 CR4: 001426e0 [ 740.783390] Stack: [ 740.792577] 812e8dcd 0048 0002 882f908c8468 [ 740.827003] 01bef000 882f928e4600 882f28c23e48 882f28c23d70 [ 740.860971] 882f28c23d38 812e8dcd 0001 882f908c8300 [ 740.894994] Call Trace: [ 740.906211] [] ? fuse_direct_IO+0xdd/0x280 [ 740.932940] [] fuse_direct_IO+0xdd/0x280 [ 740.958866] [] generic_file_direct_write+0x9e/0x150 [ 740.989318] [] fuse_file_write_iter+0x15c/0x2e0 [ 741.017725] [] __vfs_write+0xa7/0xf0 [ 741.041787] [] vfs_write+0xa9/0x190 [ 741.065307] [] SyS_pwrite64+0x69/0xa0 [ 741.090141] [] ? SyS_rt_sigprocmask+0x67/0xb0 [ 741.135924] [] entry_SYSCALL_64_fastpath+0x16/0x75 [ 741.183478] Code: 4c 03 05 32 d8 e3 7e 4d 8b 30 49 8b 40 10 4d 85 f6 0f 84 22 01 00 00 48 85 c0 0f 84 19 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 06 4c 89 f0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 [ 741.306817] RIP [] kmem_cache_alloc_trace+0x7a/0x1f0 The problem has also been documented by somebody else in the Fedora bug tracker at https://bugzilla.redhat.com/show_bug.cgi?id=1254310 This behaviour is 100% reproducible. I have asked the fuse-devel mailinglist for advice, but up to this point with no success: http://sourceforge.net/p/fuse/mailman/message/34537139/ We are still investigating if this issue is also happening with 4.0 and will add the information to this bug report once we have it. Any help on debugging will be greatly appreciated. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :