On 07/01/2012 07:23 AM, Darren Reed wrote: > On 1/07/2012 8:35 AM, Sašo Kiselkov wrote: >> On 06/29/2012 05:28 PM, Darren Reed wrote: >>> On 30/06/2012 1:10 AM, Darren Reed wrote: >>>> This patch won't work for UDP if the udp ring count >>>> is greater than the TCP ring count. This is because the >>>> code in mac_rx_srs_fanout() limits the looping over >>>> the enqueued data structures to that of "fanout", >>>> which is only set once. >>>> >>>> The bug is the loop at the bottom of mac_rx_srs_fanout(). >>>> >>>> There are a few things that could be done with it... >>> Of course I forgot to read this.. >>> >>> * Since the softrings are never destroyed and we always >>> * create equal number of softrings for TCP, UDP and rest, >>> * its OK to check one of them for count and use it without >>> * any lock. In future, if soft rings get destroyed because >>> * of reduction in fanout, we will need to ensure that happens >>> * behind the SRS_PROC. >>> >>> ... so there's a separate ring count for tcp, udp and other, >>> yet all three counters are the same but used like they are >>> different when the code thinks that it is ok to be like that. >>> >>> Is one of joyent/nexenta/someone looking at this as part >>> of a (major) project? >> So I progressed with the issue and I can see that in the correct course >> of events, I see this call bad flow in the interrupt handler (when >> fanout is handled by multiple CPUs): >> >> CPU FUNCTION >> 0 -> mac_rx_srs_drain >> 0 -> mac_rx_srs_fanout >> 0 -> mac_rx_srs_long_fanout >> 0 <- mac_rx_srs_long_fanout >> 0 -> mac_rx_soft_ring_process >> 0 -> ip_input >> 0 -> ip_drop_input >> 0 | ip_drop_input:entry str: "ipIfStatsInHdrErrors" >> 0 <- ip_drop_input >> 0 <- ip_input >> 0 <- mac_rx_soft_ring_process >> 0 <- mac_rx_srs_fanout >> 0 <- mac_rx_srs_drain >> >> It would seem that once multi-CPU fanout is enabled, multicast packets >> get corrupted somewhere in the processing, which leads ip_input >> (actually an inlined ip_input_common_v4) to throw them away (exactly at >> ip_input.c:319). > > In this case, it would appear that it isn't the IP header that is > corrupted but rather the mblk_t - it hasn't been prepared for > transmission into IP by setting b_rptr to the start of the IP > header. What's required is to adjust b_rptr. > > In mac_sched.c, after each call to mac_rx_srs_long_fanout(), > you need to add a line like this: > mp->b_rptr += hdrsize; > immediately before the FANOUT_ENQUEUE_MP().
I did and indeed it fixed the ip_drop_input issue, however, I think something broken because of it, because now I can see that my machine stops responding to IGMP requests from the router as soon as I switch this code path on. That means that while packets do fan out to multiple cores for IP processing, the upstream router cuts of me off after a while because it stops seeing IGMP membership reports from me (since at input the IGMP queries from the router get filtered, my machine never responds to them, leading the router to conclude that I must have lost interest in the multicast group). I'm still trying to determine exactly why this is happening... I'll try to dtrace around to see where in the IP layer the packets get lost. All the best, -- Saso ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
