Source: linux Version: 3.16.39-1 Tags: patch Hi,
As discussed via IRC, after upgrading to the latest update of linux-3.16, I've noticed a BUG where a null pointer is dereferenced in sunrpc. This then leads to a system stall and the NMI watchdog being triggered. Ben suggested that a lock is probably held by the core that triggered the BUG. Commit 1cded9d[fix] in master appears to be related. After deploying a new kernel with that patch, the BUG hasn't been triggered since. FWIW, it appears that some stable branches have picked up that patch already. [fix] https://github.com/torvalds/linux/commit/1cded9d The Oops and the backtrace of the core that triggered it follows: [ 159.342383] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [ 159.342388] IP: [<ffffffffa131ab67>] rpc_pipe_read+0xf7/0x150 [sunrpc] [ 159.342397] PGD 48bb5d067 PUD 4881e4067 PMD 0 [ 159.342400] Oops: 0002 [#1] SMP [ 159.342402] Modules linked in: nfsv3 nfsv4 dns_resolver vmnet(O) bnep binfmt_misc vmw_vsock_vmci_transport vsock vmw_vmci vmmon(O) iptable_nat nf_nat_ipv4 ipt_REJECT xt_policy iptable_mangle iptable_ raw nf_conntrack_ipv4 nf_defrag_ipv4 ipt_ULOG nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda xt_recent ip6table_nat nf_nat_ipv6 nf_nat xt_comment ip6t_REJECT xt_addrt ype xt_mark ip6table_mangle nf_conntrack_snmp xt_tcpudp xt_CT ip6table_raw xt_multiport nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack xt_NFLOG nfnetlink_log xt_LOG nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane nf_conntrack_proto_u dplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp nf_conntrack_amanda nf_connt rack rpcsec_gss_krb5 nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc iptable_filter ip_tables ip6table_filter ip6_tables x_tables deflate ctr twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_co mmon camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 xts serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_avx_x86_64 cast5_generic c ast_common des_generic cbc cmac xcbc rmd160 sha512_ssse3 sha512_generic hmac crypto_null af_key xfrm_algo algif_skcipher af_alg sha256_ssse3 sha256_generic dm_crypt nls_utf8 nls_cp437 vfat fat x86_pkg_temp_thermal intel_powerclamp intel_ rapl coretemp kvm_intel kvm hid_generic uvcvideo videobuf2_vmalloc usbhid hid videobuf2_memops videobuf2_core v4l2_common videodev crc32_pclmul hp_wmi media snd_hda_codec_idt snd_hda_codec_generic snd_hda_codec_hdmi nvidia(PO) sparse_key map snd_hda_intel ppdev mxm_wmi ecb aesni_intel arc4 btusb bluetooth snd_hda_controller hp_accel aes_x86_64 efi_pstore lrw gf128mul iTCO_wdt iTCO_vendor_support iwldvm mac80211 iwlwifi lis3lv02d glue_helper evdev tpm_infineon ablk_helper pcspkr 6lowpan_iphc joydev input_polldev cfg80211 snd_hda_codec cryptd serio_raw snd_hwdep drm rtsx_pci_ms mei_me tpm_tis memstick snd_pcm efivars wmi parport_pc parport rfkill snd_timer mei tpm snd soundcore i2c_i801 processor video i2 c_core button battery lpc_ich ie31200_edac edac_core shpchp ac hp_wireless fuse autofs4 ext4 crc16 mbcache jbd2 dm_mod sg sd_mod sr_mod crc_t10dif cdrom crct10dif_generic rtsx_pci_sdmmc mmc_core crct10dif_pclmul crct10dif_common crc32c_i ntel ahci libahci psmouse libata ehci_pci xhci_hcd ehci_hcd scsi_mod rtsx_pci mfd_core e1000e ptp usbcore pps_core usb_common thermal thermal_sys [ 159.342501] CPU: 4 PID: 1253 Comm: rpc.gssd Tainted: P O 3.16.0-4-amd64 #1 Debian 3.16.39-1+c9+1 [ 159.342503] Hardware name: Hewlett-Packard HP ZBook 15/1909, BIOS L70 Ver. 01.21 08/13/2014 [ 159.342504] task: ffff88048c800390 ti: ffff88048b840000 task.ti: ffff88048b840000 [ 159.342505] RIP: 0010:[<ffffffffa131ab67>] [<ffffffffa131ab67>] rpc_pipe_read+0xf7/0x150 [sunrpc] [ 159.342512] RSP: 0018:ffff88048b843ee8 EFLAGS: 00010212 [ 159.342513] RAX: ffff880059cd7808 RBX: ffff880059cd7808 RCX: 0000000000000000 [ 159.342514] RDX: 0000000000000000 RSI: 0000000000c1ec90 RDI: ffff88047561fbc8 [ 159.342515] RBP: ffff88047561fb00 R08: 0000000000000000 R09: 0000000000000000 [ 159.342516] R10: 00007ffd7d6ff790 R11: 0000000000000246 R12: ffff880446314900 [ 159.342517] R13: ffff8804758f9b68 R14: 0000000000c1ec90 R15: 0000000000000800 [ 159.342519] FS: 00007f31f23eb740(0000) GS:ffff88049dd00000(0000) knlGS:0000000000000000 [ 159.342520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 159.342521] CR2: 0000000000000008 CR3: 000000048bb5e000 CR4: 00000000001407e0 [ 159.342522] Stack: [ 159.342523] ffff880446314900 0000000000c1ec90 ffff88048b843f58 0000000000000800 [ 159.342525] 00007ffd7d6ff6c4 0000000000000003 ffffffff811aa783 ffff880446314900 [ 159.342527] ffff880446314900 0000000000c1ec90 0000000000000800 00007ffd7d6ff6c4 [ 159.342529] Call Trace: [ 159.342534] [<ffffffff811aa783>] ? vfs_read+0x93/0x170 [ 159.342536] [<ffffffff811ab3b2>] ? SyS_read+0x42/0xa0 [ 159.342540] [<ffffffff8151adcd>] ? system_call_fast_compare_end+0x10/0x15 [ 159.342541] Code: 48 89 df ff 50 20 eb a1 0f 1f 40 00 48 8d bd c8 00 00 00 e8 ac fd 1f e0 48 8b 45 00 48 39 c5 74 3f 48 8b 50 08 48 8b 08 48 89 c3 <48> 89 51 08 48 89 0a 48 8b 55 10 48 89 42 08 48 89 10 48 8d 55 [ 159.342560] RIP [<ffffffffa131ab67>] rpc_pipe_read+0xf7/0x150 [sunrpc] [ 159.342565] RSP <ffff88048b843ee8> [ 159.342566] CR2: 0000000000000008 [ 159.342568] ---[ end trace 9c1435bf68d48ad7 ]--- [ 196.735514] INFO: rcu_sched self-detected stall on CPU { 0} (t=5250 jiffies g=9939 c=9938 q=1563) [ 196.735530] sending NMI to all CPUs: [ 196.735534] NMI backtrace for cpu 0 [ 196.735547] CPU: 0 PID: 5443 Comm: vmware-vmx Tainted: P D O 3.16.0-4-amd64 #1 Debian 3.16.39-1+c9+1 [ 196.735548] Hardware name: Hewlett-Packard HP ZBook 15/1909, BIOS L70 Ver. 01.21 08/13/2014 [ 196.735550] task: ffff8804461c62d0 ti: ffff880446254000 task.ti: ffff880446254000 [ 196.735551] RIP: 0010:[<ffffffff812b97d9>] [<ffffffff812b97d9>] __const_udelay+0x9/0x30 [ 196.735557] RSP: 0018:ffff88049dc03e50 EFLAGS: 00000046 [ 196.735558] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000008 [ 196.735559] RDX: 000000000092278c RSI: 0000000000000200 RDI: 0000000000418958 [ 196.735560] RBP: ffffffff81853800 R08: 000000000000000a R09: 000000000000048e [ 196.735562] R10: 0000000000000000 R11: ffff88049dc03b96 R12: 0000000000000000 [ 196.735563] R13: ffffffff818e2fe0 R14: 000000000000061b R15: ffffffff81853800 [ 196.735565] FS: 00007eff859a5740(0000) GS:ffff88049dc00000(0000) knlGS:0000000000000000 [ 196.735566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 196.735567] CR2: 00007f16c5048548 CR3: 000000043a861000 CR4: 00000000001407f0 [ 196.735569] Stack: [ 196.735570] ffffffff81047daa ffff88049dc0d6a0 ffffffff810c73fa ffffffff8181f7c0 [ 196.735572] ffffffff810c8f45 0000000000000086 ffff8804461c62d0 0000000000000000 [ 196.735574] 0000000000000000 ffff88049dc0d1a0 ffff88049dc03f68 ffffffff810d1e00 [ 196.735577] Call Trace: [ 196.735578] <IRQ> [ 196.735579] [<ffffffff81047daa>] ? arch_trigger_all_cpu_backtrace+0x10a/0x140 [ 196.735587] [<ffffffff810c73fa>] ? rcu_check_callbacks+0x42a/0x670 [ 196.735590] [<ffffffff810c8f45>] ? timekeeping_update.constprop.9+0x35/0x70 [ 196.735593] [<ffffffff810d1e00>] ? tick_sched_handle.isra.16+0x60/0x60 [ 196.735596] [<ffffffff81075fc0>] ? update_process_times+0x40/0x70 [ 196.735599] [<ffffffff810d1dc0>] ? tick_sched_handle.isra.16+0x20/0x60 [ 196.735601] [<ffffffff810d1e3c>] ? tick_sched_timer+0x3c/0x60 [ 196.735603] [<ffffffff8108c6a7>] ? __run_hrtimer+0x67/0x210 [ 196.735605] [<ffffffff8108caa9>] ? hrtimer_interrupt+0xe9/0x220 [ 196.735617] [<ffffffff8151dcab>] ? smp_apic_timer_interrupt+0x3b/0x50 [ 196.735621] [<ffffffff8151bd3d>] ? apic_timer_interrupt+0x6d/0x80 [ 196.735622] <EOI> [ 196.735624] [<ffffffff8151a928>] ? _raw_spin_lock+0x28/0x30 [ 196.735630] [<ffffffffa13a4b5d>] ? gss_setup_upcall+0x16d/0x3b0 [auth_rpcgss] [ 196.735634] [<ffffffffa13a513b>] ? gss_cred_init+0xdb/0x380 [auth_rpcgss] [ 196.735638] [<ffffffff8119132f>] ? kmem_cache_alloc_trace+0x23f/0x470 [ 196.735641] [<ffffffff810a95c0>] ? prepare_to_wait_event+0xf0/0xf0 [ 196.735650] [<ffffffffa130c5f1>] ? rpcauth_lookup_credcache+0x151/0x220 [sunrpc] [ 196.735656] [<ffffffffa130c096>] ? rpcauth_lookupcred+0x86/0xe0 [sunrpc] [ 196.735662] [<ffffffffa130cce3>] ? rpcauth_refreshcred+0x133/0x1b0 [sunrpc] [ 196.735668] [<ffffffffa1303b78>] ? xprt_lock_and_alloc_slot+0x68/0x80 [sunrpc] [ 196.735673] [<ffffffffa12ffb50>] ? call_retry_reserve+0x60/0x60 [sunrpc] [ 196.735678] [<ffffffffa12ffb50>] ? call_retry_reserve+0x60/0x60 [sunrpc] [ 196.735684] [<ffffffffa130a3a0>] ? __rpc_execute+0x90/0x460 [sunrpc] [ 196.735687] [<ffffffff810a9211>] ? wake_up_bit+0x11/0x20 [ 196.735692] [<ffffffffa1302079>] ? rpc_run_task+0x69/0x80 [sunrpc] [ 196.735698] [<ffffffffa15dd566>] ? nfs4_call_sync_sequence+0x56/0x80 [nfsv4] [ 196.735702] [<ffffffffa15e14dd>] ? _nfs4_lookup_root.isra.34+0xcd/0xe0 [nfsv4] [ 196.735705] [<ffffffff81191ef6>] ? __kmalloc+0x4c6/0x4d0 [ 196.735709] [<ffffffffa15e3d4e>] ? nfs4_lookup_root+0x4e/0xf0 [nfsv4] [ 196.735713] [<ffffffffa15e3e3c>] ? nfs4_lookup_root_sec+0x4c/0x60 [nfsv4] [ 196.735717] [<ffffffffa15e3e85>] ? nfs4_find_root_sec+0x35/0xa0 [nfsv4] [ 196.735721] [<ffffffffa15e9f66>] ? nfs4_proc_get_rootfh+0x56/0x90 [nfsv4] [ 196.735726] [<ffffffffa16004c6>] ? nfs4_get_rootfh+0x46/0x130 [nfsv4] [ 196.735731] [<ffffffffa1301be2>] ? rpc_clone_client_set_auth+0x42/0x50 [sunrpc] [ 196.735737] [<ffffffffa1600819>] ? nfs4_server_common_setup+0x99/0x1d0 [nfsv4] [ 196.735742] [<ffffffffa1601e41>] ? nfs4_create_referral_server+0xf1/0x180 [nfsv4] [ 196.735747] [<ffffffffa15fa3db>] ? nfs4_remote_referral_mount+0x6b/0xc0 [nfsv4] [ 196.735753] [<ffffffffa1381800>] ? nfs_clone_super+0x130/0x130 [nfs] [ 196.735758] [<ffffffffa13804a0>] ? nfs_show_stats+0x480/0x480 [nfs] [ 196.735761] [<ffffffff811ae374>] ? mount_fs+0x34/0x1a0 [ 196.735764] [<ffffffff811c8052>] ? vfs_kern_mount+0x62/0x110 [ 196.735769] [<ffffffffa15fa4ad>] ? nfs_do_root_mount+0x7d/0xc0 [nfsv4] [ 196.735774] [<ffffffffa15fa796>] ? nfs4_referral_mount+0x36/0xb0 [nfsv4] [ 196.735776] [<ffffffff811ae374>] ? mount_fs+0x34/0x1a0 [ 196.735778] [<ffffffff811c8052>] ? vfs_kern_mount+0x62/0x110 [ 196.735783] [<ffffffffa16000fc>] ? nfs4_submount+0x6dc/0x7b0 [nfsv4] [ 196.735788] [<ffffffffa1389c4d>] ? nfs_d_automount+0xbd/0x1b0 [nfs] [ 196.735791] [<ffffffff811b3df7>] ? follow_managed+0x127/0x2b0 [ 196.735794] [<ffffffff811b46ff>] ? lookup_fast+0x17f/0x2b0 [ 196.735799] [<ffffffffa13772f8>] ? nfs_permission+0xe8/0x1d0 [nfs] [ 196.735801] [<ffffffff811b583b>] ? link_path_walk+0x1ab/0x870 [ 196.735803] [<ffffffff811b5701>] ? link_path_walk+0x71/0x870 [ 196.735804] [<ffffffff811b5f5b>] ? path_lookupat+0x5b/0x780 [ 196.735807] [<ffffffff811b3c17>] ? complete_walk+0x47/0xd0 [ 196.735809] [<ffffffff811b66a6>] ? filename_lookup+0x26/0xc0 [ 196.735811] [<ffffffff811ba8e4>] ? user_path_at_empty+0x54/0x90 [ 196.735814] [<ffffffff810ed71e>] ? from_kgid_munged+0xe/0x20 [ 196.735816] [<ffffffff811af53a>] ? cp_new_stat+0x13a/0x160 [ 196.735818] [<ffffffff811af0e6>] ? vfs_fstatat+0x46/0x90 [ 196.735821] [<ffffffff811af5bd>] ? SYSC_newlstat+0x1d/0x40 [ 196.735824] [<ffffffff8151adcd>] ? system_call_fast_compare_end+0x10/0x15 [ 196.735825] Code: 00 00 48 ff c8 75 fb 48 ff c8 c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 8b 05 6c 0a 5e 00 ff e0 66 90 65 48 8b 14 25 e0 26 01 00 <48> 8d 0c 12 48 c1 e2 06 48 8d 04 bd 00 00 00 00 48 29 ca f7 e2 Cheers, -- Raphael Geissert - Debian Developer www.debian.org - get.debian.net
