On Wed, Jan 25, 2012 at 05:18:21PM -0500, Eddie Kohler wrote: > Hi Luigi, > > So just to check: Does #3 include our allocator fixes?
yes it does (at least in the initial version i sent you -- your code was marginally faster than mine but if i remember well the difference only showed up in the infinitesource->sink case) > It might be possible to get rid of at least one of the copies in #3. yes but i believe it might require the use of the native netmap API (unless you processg the packet to completion within pcap_dispatch()). A suggestion i have received (but not tried yet) is to make buffers (2k-64) bytes so they make a better use of the cache. Right now, with 2k netmap buffers and 64-byte cache lines, the initial part of the packet maps to 1/16 of the available cache space, and data touching ops might cause extra cache misses (i am not using explicit prefetch, so read misses are going to be there anyways; but writes could be affected). Making the buffers shorter would spread them more evenly among cache lines. I don't remember how large are the packet buffers in Click and whether this can make a difference. I was also not particularly excited to make a change just to look good in benchmarks, but perhaps there is a point because the initial part of the packet is the one with headers, so there is going to be a lot more contention there than in other parts of the buffer. cheers luigi > Eddie > > > On 1/20/12 5:12 PM, Luigi Rizzo wrote: > >On Fri, Jan 20, 2012 at 10:21:15PM +0200, Beyers Cronje wrote: > >>Hi Luigi, > >> > >>Have you guys released the driver for linux yet, or is it still a work in > >>progress? How does the libpcap emulation compare performance wise with a > >>native netmap driver? > > > >the linux driver is not ready yet, though today we managed to > >send the first packets with linux-netmap. I am hoping we can get > >something ready for distribution in 2-3 weeks. > > > >As for native vs libpcap, here are the most recent numbers i have are > >for something equivalent to FromDevice -> Queue -> ToDevice : > > > > 1. pure netmap (zero copy) 14.2 Mpps @ 1.6 GHz > > 2. pcap_dispatch(pcap_inject()) 8.5 Mpps @ 2.93 GHz > > 3. click userspace+netmap/libpcap 3.95Mpps @ 2.93 GHz > > 4. click userspace+bpf/libpcap 0.49Mpps @ 2.93 GHz > > > >In terms of data copies and memory allocations: > >#1 is really zero-copy, #2 does one copy, #3 does two copies and > >one allocation (and the allocator was the bottleneck, fixed last > >summer with Eddie), #4 is copies plus one Packet allocation plus > >one syscall plus two mbuf allocations. > > > >cheers > >luigi > > > > > > > >>Beyers > >> > >>On Fri, Jan 20, 2012 at 8:54 PM, Luigi Rizzo<[email protected]> wrote: > >> > >>>On Fri, Jan 20, 2012 at 09:19:43AM -0800, Roman Chertov wrote: > >>>>> > >>>>>Kinda wonder how prepared click is for the netmap work being done and > >>>>>10Gb performance, and if it will help > >>>i > > > >>>>I recall reading on the list that netmap maintains a TX ring and as such > >>>>it is not friendly to Click configs that can hold on to some packets for > >>>>a while. > >>>> > >>>>FromDevice(ethX) -> cl::Classifier(...)[0] > >>>> -> Queue > >>>> -> DelayShaper(DELAY) > >>>> -> outQ; > >>>> > >>>>cl[1] -> outQ; > >>>> > >>>>Any clarifications from people familiar with netmap? > >>> > >>>Click uses the libpcap emulation to run on top of netmap, > >>>so it doesn't know anything about the netmap rings. > >>>Incoming frames are copied into Click's Packets in the callback > >>>passed tp pcap_dispatch(), and they > >>>can be stored for as long as you like provided there is > >>>enough (userspace) memory. > >>>When it is time to transmit, pcap_inject() copies the payload > >>>from the Packet to the netmap ring and the packet goes out. > >>> > >>>The two copies are moderately expensive but we are still talking > >>>3-4Mpps for FromDevice -> EtherSwitch -> ToDevice . > >>> > >>>Anything that takes a timestamp is going to hurt much more > >>>than the memory copies. > >>> > >>>cheers > >>>luigi > >>>_______________________________________________ > >>>click mailing list > >>>[email protected] > >>>https://amsterdam.lcs.mit.edu/mailman/listinfo/click > >>> > >_______________________________________________ > >click mailing list > >[email protected] > >https://amsterdam.lcs.mit.edu/mailman/listinfo/click _______________________________________________ click mailing list [email protected] https://amsterdam.lcs.mit.edu/mailman/listinfo/click
