On Sun, Sep 4, 2016 at 4:57 PM, Michał Purzyński <michalpurzyns...@gmail.com> wrote: > Thank you for a detailed answer! > > 4.3.15 was the one I was looking at, so indeed - not the current one. In > 4.3.15 choosing between those two ways of copying data was done depending on > the packet split. > > ixgbe_alloc_mapped_skb() does not exist in 4.4.6 anymore. > > While surprised, I get it that with modern hardware copying some data around > might not be that bad as it sounds. We have tremendous bandwidth available > in 2016 ;) > > So ixgbe copies (with DMA) entire packet into that page and then memcpy() > from that page to skbbuf?
Yes. Although I wouldn't really think of it as a copy. The memory on the device is meant to really just act as a buffer for incoming packets so that we can hold them long enough to be able to packetize them and send them to system memory via the PCIe write transactions. > Pardon my ignorance, but what kind of memory is used on the card, that's so > expensive? I bet it's not my usual DRAM, which is crazy cheap. Something > with less latency? Just curious. I don't know the details of the memory logic used, but the memory I am referring to would be a part of the silicon itself. Having on-die memory adds to the overall cost of the Ethernet silicon, so bigger FIFO buffers would mean more expensive devices. > Interesting fact - AF_Packet with TPacket_v3 has the same zero-copy > mechanism as pf_ring - socket buffers are mapped into userspace. That's what > I use currently, with a great success. Okay so it sounds like you are already familiar with it then, good to know. > On Sun, Sep 4, 2016 at 7:20 PM, Alexander Duyck <alexander.du...@gmail.com> > wrote: >> >> On Sun, Sep 4, 2016 at 2:50 AM, Michał Purzyński >> <michalpurzyns...@gmail.com> wrote: >> > My IXGBE data path journey made me ask questions, like the one above. >> > Basically (I'm skipping a few steps here): >> > >> > 1. packet arrives to the card and is verified by MAC >> > 2. packet is placed in the card's FIFO (which is small) >> > 3. lots of steps here but finally card does DMA into one of receive >> > buffers >> > 4. are these buffers shared with skb->data ? Or does 82599 copy bits >> > between rx buffers and skb->data ? >> > >> > Given that dma_map_single(rx_ring->dev, skb->data,...) is called in >> > ixgbe_alloc_maped_skb() I would say "card does DMA into SKB" (excuse my >> > oversimplification) but I'd like to verify it, especially that >> > ixgbe_alloc_mapped_page() can be used instead, depending on >> > CONFIG_IXGBE_DISABLE_PACKET_SPLIT >> >> So what you are describing sounds like an older ixgbe driver. We used >> to write either all the data into skb->data or the packet header. If >> the Rx path is using ixgbe_alloc_mapped_skb() then yes, we are writing >> directly to skb->data. The cost for doing that though is that we have >> to leave the skb allocated prior to actually receiving the Rx buffer >> which can be pretty expensive since the skb has to reserve the space >> for the sk_buff plus the skb_shared info. >> >> Several years ago I rewrote that so that we received the entire frame >> into one or more page based buffers. After that we copy the header of >> the frame from the page and into skb->data. The advantage to that >> approach is that we don't have to map/unmap the Rx buffers from the >> device which can be very expensive when an IOMMU is enabled. >> >> > I'm trying to understand where data copying occurs, because it might be >> > impossible to go with some clever memory mappings. >> >> It depends on the driver being used. If you are using an older one >> with ixgbe_alloc_mapped_skb then the data copy will occur when you >> transition to user space. So the copy occurs up at the socket level. >> If you are using a newer driver there is a small copy in the driver, >> and then the data gets copied up at the socket level. >> >> > A side question - why is the FIFO buffer that small? What's stopping us >> > from having it (theoretically) say 1GB? Some (super expensive) capture >> > cards do this, but they have other challenges. >> >> I think you just answered your own question. Adding memory would add >> to the overall cost of the part making it more expensive. >> >> > Of course, X520 is not a capture card (but it's very efficient doing it, >> > anyway). I do not believe in a dedicated hardware anymore. >> >> If you are looking at just doing straight capture and analysis another >> option you might look at would be something like PF_RING as I believe >> it is meant to address some of what you are pointing out by avoiding >> the copy to user-space. >> >> - Alex > > ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
