On 06/29/2012 05:28 PM, Darren Reed wrote: > On 30/06/2012 1:10 AM, Darren Reed wrote: >> This patch won't work for UDP if the udp ring count >> is greater than the TCP ring count. This is because the >> code in mac_rx_srs_fanout() limits the looping over >> the enqueued data structures to that of "fanout", >> which is only set once. >> >> The bug is the loop at the bottom of mac_rx_srs_fanout(). >> >> There are a few things that could be done with it... > > Of course I forgot to read this.. > > * Since the softrings are never destroyed and we always > * create equal number of softrings for TCP, UDP and rest, > * its OK to check one of them for count and use it without > * any lock. In future, if soft rings get destroyed because > * of reduction in fanout, we will need to ensure that happens > * behind the SRS_PROC. > > ... so there's a separate ring count for tcp, udp and other, > yet all three counters are the same but used like they are > different when the code thinks that it is ok to be like that. > > Is one of joyent/nexenta/someone looking at this as part > of a (major) project?
So I progressed with the issue and I can see that in the correct course of events, I see this call bad flow in the interrupt handler (when fanout is handled by multiple CPUs): CPU FUNCTION 0 -> mac_rx_srs_drain 0 -> mac_rx_srs_fanout 0 -> mac_rx_srs_long_fanout 0 <- mac_rx_srs_long_fanout 0 -> mac_rx_soft_ring_process 0 -> ip_input 0 -> ip_drop_input 0 | ip_drop_input:entry str: "ipIfStatsInHdrErrors" 0 <- ip_drop_input 0 <- ip_input 0 <- mac_rx_soft_ring_process 0 <- mac_rx_srs_fanout 0 <- mac_rx_srs_drain It would seem that once multi-CPU fanout is enabled, multicast packets get corrupted somewhere in the processing, which leads ip_input (actually an inlined ip_input_common_v4) to throw them away (exactly at ip_input.c:319). Any ideas on why this corruption might occur. I believe that since we're passing through a different call flow here than in the single-CPU fanout case, the relevant bits should be in the differences. Here's the call flow for single-CPU fanout: CPU FUNCTION 0 -> mac_rx_srs_drain 0 -> mac_rx_srs_proto_fanout 0 -> mac_rx_soft_ring_process 0 -> mac_rx_deliver 0 -> mac_vlan_header_info 0 -> mac_header_info 0 <- mac_header_info 0 <- mac_vlan_header_info 0 -> ip_input 0 -> ill_input_short_v4 0 -> ilb_has_rules 0 <- ilb_has_rules 0 -> ire_multicast 0 -> ire_refhold 0 <- ire_refhold 0 <- ire_multicast 0 -> ire_recv_multicast_v4 0 -> ill_hasmembers_v4 0 -> ill_hasmembers_v6 0 -> ilm_lookup 0 <- ilm_lookup 0 <- ill_hasmembers_v6 0 <- ill_hasmembers_v4 0 -> ip_input_multicast_v4 0 -> ip_fanout_v4 0 -> ip_input_cksum_v4 0 <- ip_input_cksum_v4 0 -> ip_fanout_udp_multi_v4 0 -> conn_wantpacket 0 -> conn_hasmembers_ill_withsrc_v4 0 <- conn_hasmembers_ill_withsrc_v4 0 <- conn_wantpacket 0 -> ip_fanout_udp_conn 0 -> udp_input 0 -> udp_ulp_recv 0 <- udp_ulp_recv 0 <- udp_input 0 <- ip_fanout_udp_conn 0 <- ip_fanout_udp_multi_v4 0 <- ip_fanout_v4 0 <- ip_input_multicast_v4 0 <- ire_recv_multicast_v4 0 <- ill_input_short_v4 0 -> ire_refrele 0 <- ire_refrele 0 <- ip_input 0 <- mac_rx_deliver 0 <- mac_rx_soft_ring_process 0 <- mac_rx_srs_proto_fanout 0 <- mac_rx_srs_drain Cheers, -- Saso ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
