Thank you for a detailed answer! 4.3.15 was the one I was looking at, so indeed - not the current one. In 4.3.15 choosing between those two ways of copying data was done depending on the packet split.
ixgbe_alloc_mapped_skb() does not exist in 4.4.6 anymore. While surprised, I get it that with modern hardware copying some data around might not be that bad as it sounds. We have tremendous bandwidth available in 2016 ;) So ixgbe copies (with DMA) entire packet into that page and then memcpy() from that page to skbbuf? Pardon my ignorance, but what kind of memory is used on the card, that's so expensive? I bet it's not my usual DRAM, which is crazy cheap. Something with less latency? Just curious. Interesting fact - AF_Packet with TPacket_v3 has the same zero-copy mechanism as pf_ring - socket buffers are mapped into userspace. That's what I use currently, with a great success. On Sun, Sep 4, 2016 at 7:20 PM, Alexander Duyck <alexander.du...@gmail.com> wrote: > On Sun, Sep 4, 2016 at 2:50 AM, Michał Purzyński > <michalpurzyns...@gmail.com> wrote: > > My IXGBE data path journey made me ask questions, like the one above. > > Basically (I'm skipping a few steps here): > > > > 1. packet arrives to the card and is verified by MAC > > 2. packet is placed in the card's FIFO (which is small) > > 3. lots of steps here but finally card does DMA into one of receive > buffers > > 4. are these buffers shared with skb->data ? Or does 82599 copy bits > > between rx buffers and skb->data ? > > > > Given that dma_map_single(rx_ring->dev, skb->data,...) is called in > > ixgbe_alloc_maped_skb() I would say "card does DMA into SKB" (excuse my > > oversimplification) but I'd like to verify it, especially that > > ixgbe_alloc_mapped_page() can be used instead, depending on > > CONFIG_IXGBE_DISABLE_PACKET_SPLIT > > So what you are describing sounds like an older ixgbe driver. We used > to write either all the data into skb->data or the packet header. If > the Rx path is using ixgbe_alloc_mapped_skb() then yes, we are writing > directly to skb->data. The cost for doing that though is that we have > to leave the skb allocated prior to actually receiving the Rx buffer > which can be pretty expensive since the skb has to reserve the space > for the sk_buff plus the skb_shared info. > > Several years ago I rewrote that so that we received the entire frame > into one or more page based buffers. After that we copy the header of > the frame from the page and into skb->data. The advantage to that > approach is that we don't have to map/unmap the Rx buffers from the > device which can be very expensive when an IOMMU is enabled. > > > I'm trying to understand where data copying occurs, because it might be > > impossible to go with some clever memory mappings. > > It depends on the driver being used. If you are using an older one > with ixgbe_alloc_mapped_skb then the data copy will occur when you > transition to user space. So the copy occurs up at the socket level. > If you are using a newer driver there is a small copy in the driver, > and then the data gets copied up at the socket level. > > > A side question - why is the FIFO buffer that small? What's stopping us > > from having it (theoretically) say 1GB? Some (super expensive) capture > > cards do this, but they have other challenges. > > I think you just answered your own question. Adding memory would add > to the overall cost of the part making it more expensive. > > > Of course, X520 is not a capture card (but it's very efficient doing it, > > anyway). I do not believe in a dedicated hardware anymore. > > If you are looking at just doing straight capture and analysis another > option you might look at would be something like PF_RING as I believe > it is meant to address some of what you are pointing out by avoiding > the copy to user-space. > > - Alex >
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
