We are seeing openAFS client kernel crashes on a server that is using both AFS and lustre clients. This happened using openafs 1.6.5 and 1.6.6. These machines are running a lustre client version 2.1.5. The OS on the server is RHEL 6.5. We currently are not sure what is causing these crashes. (we have not moved to the 2.6.32-431.5.1 kernel since it is not offically supported by lustre).
Any help with this issue would be greatly appreciated. Please let me know if you need any additional information. Thanks for all of your help on this. ---------------- Randy Philipp [email protected] Below are the kernel trace for the two crashes: Mar 26 14:09:48 maya-usr1 kernel: openafs: Can't get dentry Mar 26 14:09:48 maya-usr1 kernel: ------------[ cut here ]------------ Mar 26 14:09:48 maya-usr1 kernel: kernel BUG at /usr/src/openafs-kernel-1.6.6/src/src/libafs/MODLOAD-2.6.32-358.23.2.el6.bright_custom.x86_64-MP/osi_file.c:53! Mar 26 14:09:48 maya-usr1 kernel: invalid opcode: 0000 [#1] SMP Mar 26 14:09:48 maya-usr1 kernel: last sysfs file: /sys/devices/system/node/node1/meminfo Mar 26 14:09:48 maya-usr1 kernel: CPU 10 Mar 26 14:09:48 maya-usr1 kernel: Modules linked in: libafs(P)(U) autofs4 mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nvidia(P)(U) i2c_core ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa sr_mod cdrom dcdbas microcode wmi power_meter ib_qib ib_mad ib_core usb_storage shpchp iTCO_wdt iTCO_vendor_support hpilo xfs exportfs isci libsas qla3xxx cciss hpsa e1000e ixgbe igb dca ptp pps_core megaraid_sas aacraid aic79xx aic7xxx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2 forcedeth ext4 jbd2 ext3 jbd ext2 mbcache sata_sil tg3 e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc tun sg sd_mod crc_t10dif ahci bnx2x libcrc32c mdio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] Mar 26 14:09:48 maya-usr1 kernel: Mar 26 14:09:48 maya-usr1 kernel: Pid: 12283, comm: ls Tainted: P --------------- 2.6.32-358.23.2.el6.bright_custom.x86_64 #1 Dell Inc. PowerEdge R720/0H5J4J Mar 26 14:09:48 maya-usr1 kernel: RIP: 0010:[<ffffffffa10403b1>] [<ffffffffa10403b1>] afs_linux_raw_open+0xa1/0x110 [libafs] Mar 26 14:09:48 maya-usr1 kernel: RSP: 0018:ffff880fe1fd97f8 EFLAGS: 00010296 Mar 26 14:09:48 maya-usr1 kernel: RAX: 0000000000000020 RBX: ffffffffffffff8c RCX: 000000000000533b Mar 26 14:09:48 maya-usr1 kernel: RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246 Mar 26 14:09:48 maya-usr1 kernel: RBP: ffff880fe1fd9808 R08: 0000000000000000 R09: ffffffff8163fde0 Mar 26 14:09:48 maya-usr1 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff880fed915000 Mar 26 14:09:48 maya-usr1 kernel: R13: ffff880f8cd63540 R14: ffffc90026dba500 R15: ffff880fe1fd9a38 Mar 26 14:09:48 maya-usr1 kernel: FS: 00002aaaab8acc40(0000) GS:ffff8800606a0000(0000) knlGS:0000000000000000 Mar 26 14:09:48 maya-usr1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Mar 26 14:09:48 maya-usr1 kernel: CR2: 00002aaaab1df000 CR3: 0000001fe840d000 CR4: 00000000001407e0 Mar 26 14:09:48 maya-usr1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 26 14:09:48 maya-usr1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Mar 26 14:09:48 maya-usr1 kernel: Process ls (pid: 12283, threadinfo ffff880fe1fd8000, task ffff880f8cd63540) Mar 26 14:09:48 maya-usr1 kernel: Stack: Mar 26 14:09:48 maya-usr1 kernel: ffffc90026dba5c8 ffff880fed915000 ffff880fe1fd9838 ffffffffa10404d4 Mar 26 14:09:48 maya-usr1 kernel: <d> 0000000000000000 ffffc9001f0c2960 0000000000000000 ffffc90026dba500 Mar 26 14:09:48 maya-usr1 kernel: <d> ffff880fe1fd9878 ffffffffa0febffc 00000000e1fd98a0 ffff880fe1fd9a08 Mar 26 14:09:48 maya-usr1 kernel: Call Trace: Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa10404d4>] osi_UFSOpen+0xb4/0x1a0 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa0febffc>] DRead+0x29c/0x4b0 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa0ffe342>] FindItem+0x62/0x170 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa0ffe868>] afs_dir_LookupOffset+0x28/0x80 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa1024b2e>] afs_lookup+0xb6e/0x1d30 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81223611>] ? avc_has_perm+0x71/0x90 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa1045fcd>] afs_linux_dentry_revalidate+0x13d/0x570 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffffa1013c62>] ? afs_FindVCache+0xe2/0x5b0 [libafs] Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81223611>] ? avc_has_perm+0x71/0x90 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8119af57>] ? __d_lookup+0xa7/0x150 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81190726>] do_lookup+0x66/0x230 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81190afd>] __link_path_walk+0x20d/0x1030 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8118f6e5>] ? path_to_nameidata+0x25/0x60 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81191baa>] path_walk+0x6a/0xe0 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81191d7b>] do_path_lookup+0x5b/0xa0 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81192a07>] user_path_at+0x57/0xa0 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8118de96>] ? page_put_link+0x16/0x20 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8118e828>] ? generic_readlink+0x98/0xc0 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8122598b>] ? dentry_has_perm+0x5b/0x80 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81186dbc>] vfs_fstatat+0x3c/0x80 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81277805>] ? _atomic_dec_and_lock+0x55/0x80 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81186f2b>] vfs_stat+0x1b/0x20 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81186f54>] sys_newstat+0x24/0x50 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8118f571>] ? path_put+0x31/0x40 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff81186a47>] ? sys_readlinkat+0x77/0xc0 Mar 26 14:09:48 maya-usr1 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b Mar 26 14:09:48 maya-usr1 kernel: Code: 13 e0 48 3d 00 f0 ff ff 49 89 c4 77 22 48 89 df e8 05 af 15 e0 5b 4c 89 e0 41 5c c9 c3 48 c7 c7 d1 9f 05 a1 31 c0 e8 b3 d7 4c e0 <0f> 0b eb fe 48 8b 35 b4 8a 03 00 65 48 8b 04 25 c0 cb 00 00 48 Mar 26 14:09:48 maya-usr1 kernel: RIP [<ffffffffa10403b1>] afs_linux_raw_open+0xa1/0x110 [libafs] Mar 26 14:09:48 maya-usr1 kernel: RSP <ffff880fe1fd97f8> Mar 26 14:09:48 maya-usr1 kernel: ---[ end trace 18c3de7d535f6467 ]--- Mar 26 14:09:48 maya-usr1 kernel: Kernel panic - not syncing: Fatal exception Mar 20 11:23:50 maya-usr1 kernel: openafs: Can't get dentry Mar 20 11:23:50 maya-usr1 kernel: ------------[ cut here ]------------ Mar 20 11:23:50 maya-usr1 kernel: kernel BUG at /root/rpmbuild/BUILD/openafs-1.6.5.1/src/libafs/MODLOAD-2.6.32-358.el6.x86_64-SP/osi_file.c:53! Mar 20 11:23:50 maya-usr1 kernel: invalid opcode: 0000 [#1] SMP Mar 20 11:23:50 maya-usr1 kernel: last sysfs file: /sys/devices/system/node/node1/meminfo Mar 20 11:23:50 maya-usr1 kernel: CPU 0 Mar 20 11:23:50 maya-usr1 kernel: Modules linked in: openafs(P)(U) autofs4 mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nvidia(P)(U) i2c_ core ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa sr_mod cdrom wmi power_meter usb_ storage shpchp ib_qib ib_mad ib_core dcdbas microcode iTCO_wdt iTCO_vendor_support hpilo xfs exportfs isci libsas qla3xxx cciss hpsa e1000e ixgbe igb dca ptp pps_core megaraid_sas aacraid aic79xx aic7xxx ata_piix m pt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2 forcedeth ext4 jbd2 ext3 jbd ext2 mbcache sata_sil tg3 e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc tun sg sd_mod crc_t10dif ahci bnx2x libcrc32c mdio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] Mar 20 11:23:50 maya-usr1 kernel: Mar 20 11:23:50 maya-usr1 kernel: Pid: 28853, comm: bash Tainted: P --------------- 2.6.32-358.23.2.el6.bright_custom.x86_64 #1 Dell Inc. PowerEdge R720/0H5J4J Mar 20 11:23:50 maya-usr1 kernel: RIP: 0010:[<ffffffffa10543b1>] [<ffffffffa10543b1>] afs_linux_raw_open+0xa1/0x110 [openafs] Mar 20 11:23:50 maya-usr1 kernel: RSP: 0018:ffff880fdd9b9cd8 EFLAGS: 00010292 Mar 20 11:23:50 maya-usr1 kernel: RAX: 0000000000000020 RBX: ffffffffffffff8c RCX: 00000000000053d7 Mar 20 11:23:50 maya-usr1 kernel: RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246 Mar 20 11:23:50 maya-usr1 kernel: RBP: ffff880fdd9b9ce8 R08: 0000000000000000 R09: ffffffff8163fde0 Mar 20 11:23:50 maya-usr1 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff881fe9977000 Mar 20 11:23:50 maya-usr1 kernel: R13: ffff880fdea84080 R14: ffff880fdea84080 R15: 0000000000000080 Mar 20 11:23:50 maya-usr1 kernel: FS: 00002aaaaaac7580(0000) GS:ffff880060600000(0000) knlGS:0000000000000000 Mar 20 11:23:50 maya-usr1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 20 11:23:50 maya-usr1 kernel: CR2: 0000000000921000 CR3: 00000010245ba000 CR4: 00000000001407f0 Mar 20 11:23:50 maya-usr1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Mar 20 11:23:50 maya-usr1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Mar 20 11:23:50 maya-usr1 kernel: Process bash (pid: 28853, threadinfo ffff880fdd9b8000, task ffff880fdea84080) Mar 20 11:23:50 maya-usr1 kernel: Stack: Mar 20 11:23:50 maya-usr1 kernel: ffffc900253c45c8 ffff881fe9977000 ffff880fdd9b9d18 ffffffffa10544d4 Mar 20 11:23:50 maya-usr1 kernel: <d> 0000000000000002 ffffc9001e81de10 0000000000001000 ffffc900253c4500 Mar 20 11:23:50 maya-usr1 kernel: <d> ffff880fdd9b9d58 ffffffffa0ffff6c ffff880fdd9b9d48 ffffffff8119c5b8 Mar 20 11:23:50 maya-usr1 kernel: Call Trace: Mar 20 11:23:50 maya-usr1 kernel: [<ffffffffa10544d4>] osi_UFSOpen+0xb4/0x1a0 [openafs] Mar 20 11:23:50 maya-usr1 kernel: [<ffffffffa0ffff6c>] DRead+0x29c/0x4b0 [openafs] Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff8119c5b8>] ? igrab+0x48/0x50 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff811965ab>] ? filldir+0x5b/0xe0 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffffa1012128>] afs_dir_GetBlob+0x18/0x40 [openafs] Mar 20 11:23:50 maya-usr1 kernel: [<ffffffffa103d120>] BlobScan+0x30/0x130 [openafs] Mar 20 11:23:50 maya-usr1 kernel: [<ffffffffa10577a7>] afs_linux_readdir+0x6f7/0xa80 [openafs] Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff81196550>] ? filldir+0x0/0xe0 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff81196550>] ? filldir+0x0/0xe0 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff811967d0>] vfs_readdir+0xc0/0xe0 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff81196959>] sys_getdents+0x89/0xf0 Mar 20 11:23:50 maya-usr1 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b Mar 20 11:23:50 maya-usr1 kernel: Code: 12 e0 48 3d 00 f0 ff ff 49 89 c4 77 22 48 89 df e8 05 6f 14 e0 5b 4c 89 e0 41 5c c9 c3 48 c7 c7 55 de 06 a1 31 c0 e8 b3 97 4b e0 <0f> 0b eb fe 48 8b 35 d4 85 03 00 65 48 8b 04 25 c0 cb 00 00 48 Mar 20 11:23:50 maya-usr1 kernel: RIP [<ffffffffa10543b1>] afs_linux_raw_open+0xa1/0x110 [openafs] Mar 20 11:23:50 maya-usr1 kernel: RSP <ffff880fdd9b9cd8> Mar 20 11:23:50 maya-usr1 kernel: ---[ end trace c6d223ec7d3ca602 ]---
