Hi, Dale, In addition to the ARP problems (thanks for the link, Adam!), you can also run into "reverse path filtering" problems. Most linux distros configure the network interfaces to perform "reverse path filtering" (rp_filter). When rp_filter is enabled for a network interface, the kernel will check whether sending a hypothetical return packet to the source IP address of a received packet would route out the same interface on which the packet was received. Packets that fail this check are known as "martians" (I did not make that up) and will be dropped if rp_filter is enabled. rp_filter (not to be confused with arp_filter) can be controlled via sysctl. Search for "ip-sysctl.txt" in the Linux kernel Documentation directory for more details.
One trick that can sometimes be used to avoid the ARP problems is to configure the 10 GbE interfaces on your computers to use a smaller than 8 bit subnet (i.e. a netmask with more ones than 255.255.255.0). This generally requires turning off rp_filter and has some other limitations, but can be useful if the data flow is ROACH-to-PC and not the other way around. Hope this helps, Dave On Jun 14, 2013, at 10:05 AM, Adam Barta wrote: > Hi > > I guess this topic is almost closed but the effect is known as ARP flux and > this article gives quite a detailed overview of the problem and two possible > solutions using sysclt. > > http://linux-ip.net/html/ether-arp.html#ex-ether-arp-overview > > Hope it helps > > > Regards > > Adam > > On Jun 14, 2556 BE, at 3:47 PM, "Gary, Dale E." <[email protected]> wrote: > >> > In particular the kernel may answer arp requests for one interface on the >> > other interface >> >> This was exactly what we were seeing. The tap_start command was not >> working, so I manually set the ARP table, but with tap_start still running >> the kernel would set both IPs (on the same subnet) to the same MAC address, >> so I had to issue a tap_stop to avoid it. Now it makes sense. >> >> I have now configured the two interfaces to have different subnets, and now >> I can receive packets on both. Thanks! >> >> I was thinking of the ROACHes as occupying a single LAN, but would a >> reasonable configuration be for ROACH-to-ROACH packets to be on, say >> 10.0.0.x, and ROACH-to-computer packets to have different subnets, e.g. >> 10.0.1.x and 10.0.2.x? >> >> Thanks, >> Dale >> >> >> On Fri, Jun 14, 2013 at 1:09 PM, John Ford <[email protected]> wrote: >> >> I am having a problem receiving 10gbe packets from one of the interfaces >> >> on >> >> a dual-port Myricom NIC. I believe the packets are properly addressed, >> >> and >> >> wireshark sees them fine, but programmatically we cannot receive them on >> >> 10.0.0.102 via C or Python (recvfrom() just hangs), while on 10.0.0.101 >> >> everything is working fine. >> > >> > Having two IPs on the same subnet can yield interesting results under >> > linux, >> > In particular the kernel may answer arp requests for one interface on the >> > other interface ... >> > >> >> Yes, we found that it doesn't work easily. I don't know the details, but >> our sysadmin advised against it. We created different 10.x.x.x subnets to >> use both NICs in one box, which worked fine. >> >> I would think there's some advanced magic that could be brought to bear, >> but we didn't find it. >> >> John >> >> > regards >> > >> > marc >> > >> >> >>
