Re: tbl->lock not taken in neigh_lookup() ?
From: Ani Sinha Date: Mon, 25 Jan 2016 10:11:15 +0530 > Can I get some insights into this? I am sure I am missing something. The whole point of RCU locking is that read accesses in the fast paths (lookups) do not need to take the spinlock. Proper RCU barriers, RCU deferred freeing of objects, and other things make it safe.
Re: tbl->lock not taken in neigh_lookup() ?
On Mon, 2016-01-25 at 10:11 +0530, Ani Sinha wrote: > Hi All: > > Can I get some insights into this? I am sure I am missing something. > > thanks > ani > > > On Thu, Jan 21, 2016 at 9:05 PM, Ani Sinha wrote: > > hi guys > > > > As per the comment at the top of net/core/neighbor.c we should be > > taking this lock even for scanning the hash buckets. I do see that > > this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i > > missing something? > > > > For the context I am investigating the following crash which happened > > on one of our boxes. This is in 3.18.19 kernel where the following > > code in neigh_lookup() returns a corrupt value of neighbour* : > > > > for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]); > > neigh_lookup() encloses its loop in rcu_read_lock_bh() .. rcu_read_unlock_bh(); So it uses RCU BH locking. > > > > > > [ 4918.117044] BUG: unable to handle kernel paging request at > > 14000270 > > [ 4918.200203] IP: [] neigh_lookup+0x64/0xdf > > [ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0 > > [ 4918.266795] Oops: [#1] PREEMPT SMP > > [ 4918.266798] Modules linked in: macvlan l2mod_dma(PO) > > arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6 > > nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4 > > nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl > > xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp > > iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr > > kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O) > > plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables > > xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables > > x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm > > [last unloaded: plxnt_ll] > > [ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 > > #1 > > [ 4918.266840] task: 8803c6dd3720 ti: 88007b98c000 task.ti: > > 88007b98c000 > > [ 4918.266841] RIP: 0010:[] [] > > neigh_lookup+0x64/0xdf > > [ 4918.266847] RSP: 0018:88007b98faa8 EFLAGS: 00210206 > > [ 4918.266849] RAX: 8803203bf278 RBX: 81866990 RCX: > > 0014 > > [ 4918.266851] RDX: 88032a46624c RSI: 88005ff82000 RDI: > > 880319ee6020 > > [ 4918.266852] RBP: 88007b98fad8 R08: 0a00 R09: > > 880319ee6000 > > [ 4918.266854] R10: 88007b98faf8 R11: 88043d001900 R12: > > 14000100 > > [ 4918.266855] R13: 880319ee6020 R14: 88005ff82000 R15: > > 0010 > > [ 4918.266857] FS: () GS:88044f54(0063) > > knlGS:f73cb8e0 > > [ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 80050033 > > [ 4918.266860] CR2: 14000270 CR3: 7e371000 CR4: > > 07e0 > > [ 4918.266862] Stack: > > [ 4918.266863] 0180 8185cd00 81866990 > > 880319ee6010 > > [ 4918.266866] 880319ee601c 88005ff82000 88007b98fb28 > > 813b97c2 > > [ 4918.266869] 88007b98faf8 818282c0 88007b98fb18 > > 880377411600 > > [ 4918.266872] Call Trace: > > [ 4918.266875] [] neigh_delete+0x113/0x17f > > [ 4918.266878] [] rtnetlink_rcv_msg+0x18a/0x1a0 > > [ 4918.266882] [] ? get_parent_ip+0x11/0x42 > > [ 4918.266885] [] ? rhashtable_lookup_compare+0x4b/0x71 > > [ 4918.266887] [] ? rtnetlink_rcv_msg+0x0/0x1a0 > > [ 4918.266890] [] netlink_rcv_skb+0x3e/0x94 > > [ 4918.266891] [] rtnetlink_rcv+0x21/0x28 > > [ 4918.266893] [] netlink_unicast+0x10b/0x1ad > > [ 4918.266895] [] netlink_sendmsg+0x2e6/0x327 > > [ 4918.266898] [] sock_sendmsg+0x6d/0x86 > > [ 4918.266901] [] ? sock_poll+0x10e/0x11c > > [ 4918.266903] [] ? sockfd_lookup_light+0x12/0x5d > > [ 4918.266906] [] SyS_sendto+0xf3/0x11b > > [ 4918.266909] [] ? __mutex_lock_slowpath+0x2de/0x2fe > > [ 4918.266911] [] ? get_parent_ip+0x11/0x42 > > [ 4918.266914] [] SyS_send+0xf/0x11 > > [ 4918.266916] [] compat_SyS_socketcall+0x122/0x1e3 > > [ 4918.266919] [] sysenter_dispatch+0x7/0x1e > > [ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18 > > b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c > > 8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c > > 89 fa > > [ 4918.266947] RIP [] neigh_lookup+0x64/0xdf > > [ 4918.334502] RSP > > [ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4 > > 12:34:37 PST 2016 > > > > [ 4918.455186] CR2: 14000270 You might try to reproduce the bug on linux-4.4. 3.18.19 is rather old.
Re: tbl->lock not taken in neigh_lookup() ?
Hi All: Can I get some insights into this? I am sure I am missing something. thanks ani On Thu, Jan 21, 2016 at 9:05 PM, Ani Sinha wrote: > hi guys > > As per the comment at the top of net/core/neighbor.c we should be > taking this lock even for scanning the hash buckets. I do see that > this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i > missing something? > > For the context I am investigating the following crash which happened > on one of our boxes. This is in 3.18.19 kernel where the following > code in neigh_lookup() returns a corrupt value of neighbour* : > > for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]); > > > > [ 4918.117044] BUG: unable to handle kernel paging request at 14000270 > [ 4918.200203] IP: [] neigh_lookup+0x64/0xdf > [ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0 > [ 4918.266795] Oops: [#1] PREEMPT SMP > [ 4918.266798] Modules linked in: macvlan l2mod_dma(PO) > arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6 > nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4 > nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl > xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp > iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr > kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O) > plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables > xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables > x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm > [last unloaded: plxnt_ll] > [ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 #1 > [ 4918.266840] task: 8803c6dd3720 ti: 88007b98c000 task.ti: > 88007b98c000 > [ 4918.266841] RIP: 0010:[] [] > neigh_lookup+0x64/0xdf > [ 4918.266847] RSP: 0018:88007b98faa8 EFLAGS: 00210206 > [ 4918.266849] RAX: 8803203bf278 RBX: 81866990 RCX: > 0014 > [ 4918.266851] RDX: 88032a46624c RSI: 88005ff82000 RDI: > 880319ee6020 > [ 4918.266852] RBP: 88007b98fad8 R08: 0a00 R09: > 880319ee6000 > [ 4918.266854] R10: 88007b98faf8 R11: 88043d001900 R12: > 14000100 > [ 4918.266855] R13: 880319ee6020 R14: 88005ff82000 R15: > 0010 > [ 4918.266857] FS: () GS:88044f54(0063) > knlGS:f73cb8e0 > [ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 80050033 > [ 4918.266860] CR2: 14000270 CR3: 7e371000 CR4: > 07e0 > [ 4918.266862] Stack: > [ 4918.266863] 0180 8185cd00 81866990 > 880319ee6010 > [ 4918.266866] 880319ee601c 88005ff82000 88007b98fb28 > 813b97c2 > [ 4918.266869] 88007b98faf8 818282c0 88007b98fb18 > 880377411600 > [ 4918.266872] Call Trace: > [ 4918.266875] [] neigh_delete+0x113/0x17f > [ 4918.266878] [] rtnetlink_rcv_msg+0x18a/0x1a0 > [ 4918.266882] [] ? get_parent_ip+0x11/0x42 > [ 4918.266885] [] ? rhashtable_lookup_compare+0x4b/0x71 > [ 4918.266887] [] ? rtnetlink_rcv_msg+0x0/0x1a0 > [ 4918.266890] [] netlink_rcv_skb+0x3e/0x94 > [ 4918.266891] [] rtnetlink_rcv+0x21/0x28 > [ 4918.266893] [] netlink_unicast+0x10b/0x1ad > [ 4918.266895] [] netlink_sendmsg+0x2e6/0x327 > [ 4918.266898] [] sock_sendmsg+0x6d/0x86 > [ 4918.266901] [] ? sock_poll+0x10e/0x11c > [ 4918.266903] [] ? sockfd_lookup_light+0x12/0x5d > [ 4918.266906] [] SyS_sendto+0xf3/0x11b > [ 4918.266909] [] ? __mutex_lock_slowpath+0x2de/0x2fe > [ 4918.266911] [] ? get_parent_ip+0x11/0x42 > [ 4918.266914] [] SyS_send+0xf/0x11 > [ 4918.266916] [] compat_SyS_socketcall+0x122/0x1e3 > [ 4918.266919] [] sysenter_dispatch+0x7/0x1e > [ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18 > b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c > 8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c > 89 fa > [ 4918.266947] RIP [] neigh_lookup+0x64/0xdf > [ 4918.334502] RSP > [ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4 > 12:34:37 PST 2016 > > [ 4918.455186] CR2: 14000270
tbl->lock not taken in neigh_lookup() ?
hi guys As per the comment at the top of net/core/neighbor.c we should be taking this lock even for scanning the hash buckets. I do see that this lock is taken in pneigh_lookup() but not in neigh_lookup(). Am i missing something? For the context I am investigating the following crash which happened on one of our boxes. This is in 3.18.19 kernel where the following code in neigh_lookup() returns a corrupt value of neighbour* : for (n = rcu_dereference_bh(nht->hash_buckets[hash_val]); [ 4918.117044] BUG: unable to handle kernel paging request at 14000270 [ 4918.200203] IP: [] neigh_lookup+0x64/0xdf [ 4918.266790] PGD 7e383067 PUD 7e202067 PMD 0 [ 4918.266795] Oops: [#1] PREEMPT SMP [ 4918.266798] Modules linked in: macvlan l2mod_dma(PO) arptable_filter arp_tables nf_log_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6t_REJECT nf_reject_ipv6 ip6table_mangle nf_log_ipv4 nf_log_common nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_limit xt_hl xt_conntrack ipt_REJECT nf_reject_ipv4 xt_multiport xt_tcpudp iptable_mangle sch_prio strata_dma(PO) arista_strata_bde(PO) msr kbfd(O) 8021q garp stp llc tun scd_em_driver(O) plxnt_nl(O) plxnt_ll(O) nf_conntrack_tftp iptable_raw iptable_filter ip_tables xt_CT nf_conntrack xt_mark ip6table_raw ip6table_filter ip6_tables x_tables coretemp microcode intel_ips scd(O) sb_e3_edac kvm_intel kvm [last unloaded: plxnt_ll] [ 4918.266837] CPU: 5 PID: 3315 Comm: Arp Tainted: P O 3.18.19 #1 [ 4918.266840] task: 8803c6dd3720 ti: 88007b98c000 task.ti: 88007b98c000 [ 4918.266841] RIP: 0010:[] [] neigh_lookup+0x64/0xdf [ 4918.266847] RSP: 0018:88007b98faa8 EFLAGS: 00210206 [ 4918.266849] RAX: 8803203bf278 RBX: 81866990 RCX: 0014 [ 4918.266851] RDX: 88032a46624c RSI: 88005ff82000 RDI: 880319ee6020 [ 4918.266852] RBP: 88007b98fad8 R08: 0a00 R09: 880319ee6000 [ 4918.266854] R10: 88007b98faf8 R11: 88043d001900 R12: 14000100 [ 4918.266855] R13: 880319ee6020 R14: 88005ff82000 R15: 0010 [ 4918.266857] FS: () GS:88044f54(0063) knlGS:f73cb8e0 [ 4918.266859] CS: 0010 DS: 002b ES: 002b CR0: 80050033 [ 4918.266860] CR2: 14000270 CR3: 7e371000 CR4: 07e0 [ 4918.266862] Stack: [ 4918.266863] 0180 8185cd00 81866990 880319ee6010 [ 4918.266866] 880319ee601c 88005ff82000 88007b98fb28 813b97c2 [ 4918.266869] 88007b98faf8 818282c0 88007b98fb18 880377411600 [ 4918.266872] Call Trace: [ 4918.266875] [] neigh_delete+0x113/0x17f [ 4918.266878] [] rtnetlink_rcv_msg+0x18a/0x1a0 [ 4918.266882] [] ? get_parent_ip+0x11/0x42 [ 4918.266885] [] ? rhashtable_lookup_compare+0x4b/0x71 [ 4918.266887] [] ? rtnetlink_rcv_msg+0x0/0x1a0 [ 4918.266890] [] netlink_rcv_skb+0x3e/0x94 [ 4918.266891] [] rtnetlink_rcv+0x21/0x28 [ 4918.266893] [] netlink_unicast+0x10b/0x1ad [ 4918.266895] [] netlink_sendmsg+0x2e6/0x327 [ 4918.266898] [] sock_sendmsg+0x6d/0x86 [ 4918.266901] [] ? sock_poll+0x10e/0x11c [ 4918.266903] [] ? sockfd_lookup_light+0x12/0x5d [ 4918.266906] [] SyS_sendto+0xf3/0x11b [ 4918.266909] [] ? __mutex_lock_slowpath+0x2de/0x2fe [ 4918.266911] [] ? get_parent_ip+0x11/0x42 [ 4918.266914] [] SyS_send+0xf/0x11 [ 4918.266916] [] compat_SyS_socketcall+0x122/0x1e3 [ 4918.266919] [] sysenter_dispatch+0x7/0x1e [ 4918.266920] Code: 00 00 4c 89 f6 4c 89 ef 49 8d 54 24 0c ff 53 18 b9 20 00 00 00 41 2b 4c 24 08 d3 e8 89 c0 48 c1 e0 03 49 03 04 24 4c 8b 20 eb 55 <4d> 3b b4 24 70 01 00 00 75 47 49 8d bc 24 78 01 00 00 4c 89 fa [ 4918.266947] RIP [] neigh_lookup+0x64/0xdf [ 4918.334502] RSP [ 4918.334505] Kernel version: 3.18.19 #1 SMP PREEMPT Mon Jan 4 12:34:37 PST 2016 [ 4918.455186] CR2: 14000270
