Alexander Duyck writes: > On Mon, Oct 16, 2017 at 3:53 AM, Lluís Vilanova <vilan...@ac.upc.edu> wrote: >> Alexander Duyck writes: >> >>> On Fri, Oct 13, 2017 at 11:26 PM, Lluís Vilanova <vilan...@ac.upc.edu> >>> wrote: >>>> Hi! I'm not sure if this is the right venue for this question, so feel >>>> free to >>>> redirect me. >>>> >>>> I'm trying to put some "additional information" on client-generated >>>> packets to >>>> direct them to specific queues of a X540-AT2 nic (server running on Linux); >>>> i.e., some type of client-directed queue balancing. >>>> >>>> So far, I've tried the following with no luck (any ideas?): >>>> >>>> * VLAN tag: >>>> Assigning separate VLAN tags for each target queue has as a result that my >>>> server receives no packets, since it's listening on the main NIC interface >>>> which has no VLAN tags assigned. Is there a way to drop them once >>>> received, so >>>> that they'll only be used for "queue balancing"? >>>> >> >>> So how many different queues do you need to divide your traffic into? >>> If you can work with only 8 total classification types you might look >>> at using the VLAN priority field and just use VLAN 0 for all your >>> traffic. That would work around the redirecting to VLANs issue. >> >> Unfortunately I need 16 queues (one per core on the server host). >> >> >>>> * TOS field (bits 8-15 of IPv4 header): >>>> When I use ethtool to direct packets to each queue based on the TOS field, >>>> the >>>> nic seems to ignore it (TOS is always 0 on the rules): >>>> >>>> $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10 >>>> Added rule with ID 2045 >>>> $ ethtool --show-ntuple em2 >>>> 16 RX rings available >>>> Total 1 rules >>>> Filter: 2045 >>>> Rule Type: TCP over IPv4 >>>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> TOS: 0x0 mask: 0xff >>>> Src port: 0 mask: 0xffff >>>> Dest port: 0 mask: 0xffff >>>> VLAN EtherType: 0x0 mask: 0xffff >>>> VLAN: 0x0 mask: 0xffff >>>> User-defined: 0x0 mask: 0xffffffffffffffff >>>> Action: Direct to queue 10 >>>> >> >>> Not surprising as the hardware doesn't support the TOS field for >>> filtering. The fact that it was overlooked was a bug though. You could >>> probably file a bug for that at e1000.sf.net >> >> True, I was looking at the wrong datasheet and thought TOS filtering was >> available. >> >> >>>> * user-def: >>>> I've also tried with user-def to overcome the TOS field "problem", but it >>>> seems I can only use that on the last two bytes: >>>> >>>> $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10 >>>> Added rule with ID 2045 >>>> $ ethtool --show-ntuple em2 >>>> 16 RX rings available >>>> Total 1 rules >>>> >>>> Filter: 2045 >>>> Rule Type: TCP over IPv4 >>>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> TOS: 0x0 mask: 0xff >>>> Src port: 0 mask: 0xffff >>>> Dest port: 0 mask: 0xffff >>>> VLAN EtherType: 0x0 mask: 0xffff >>>> VLAN: 0x0 mask: 0xffff >>>> User-defined: 0x2 mask: 0xffffffffffffff00 >>>> Action: Direct to queue 10 >>>> >> >>> I'm not sure what you were hoping to do with the user-defined field. >>> Last I knew the value was just used to indicate what VF that traffic >>> was going to be routed to. >> >> I thought it would map into what the NIC's datasheet describes as: >> >> "Flexible 2-byte tuple anywhere in the first 64 bytes of the packet" >> >> So I was hoping to use it as a substitute to the TOS filter (which you told >> me >> is not available on my specific NIC).
> So the way the filter works is that it will only do a 2 byte tuple at > a fixed offset based on the start of the Ethernet frame. This is > somewhat limiting since a VLAN can shift that offset by 4 bytes so you > cannot reliably use the offset value outside of the Ethernet header. > That was one of the reasons why we had hard coded the offset to be the > EtherType value that I believe is reported as VLAN EtherType since it > will either be the EtherType for the inner frame, or the EtherType of > the VLAN in the case of a VLAN being present. > You could modify the driver to make the field fit what you want, but > the limitation is that you would then have to make certain no VLAN > tags are present on the frames, or encode the rule so that a certain > VLAN tag must always be present when doing the filtering. Mmmm, that could work. I can ensure all received packets have no VLAN tags, and set a filter that checks for the IPv4 ethertype and the TOS field using the 2-byte flexible match (provided that IPv4 can be checked without using the flexible match). For the record, the flexible match offset is 6 words, which according to the comments in the code points to the ethertype field. >>>> And when I try to match some other bytes, it's simply ignored >>>> (user-defined is >>>> always zero and the mask is full): >>>> >>>> $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff >>>> action 10 >>>> Added rule with ID 2045 >>>> $ ethtool --show-ntuple em2 >>>> 16 RX rings available >>>> Total 1 rules >>>> >>>> Filter: 2045 >>>> Rule Type: TCP over IPv4 >>>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>>> TOS: 0x0 mask: 0xff >>>> Src port: 0 mask: 0xffff >>>> Dest port: 0 mask: 0xffff >>>> VLAN EtherType: 0x0 mask: 0xffff >>>> VLAN: 0x0 mask: 0xffff >>>> User-defined: 0x0 mask: 0xffffffffffffffff >>>> Action: Direct to queue 10 >>>> >>>> >>>> Thanks! >>>> Lluis >> >>> The filters for the ixgbe driver are pretty limited. As I recall you >>> have an EtherType mask, VLAN tag, VLAN priority, source IP, >>> destination IP, source port, destination port, and if it is a TCP over >>> IPv4 or UDP over IPv4 rule. Like I said I think your best chance might >>> be to look at encoding and filtering on VLAN priority but that only >>> gives you about 8 queues you could target since that is composed of >>> only 3 bits. >> >>> I hope that helps. >> >> Thanks a lot. If I can't use the "user-defined" filters I will see if I can >> hack >> into the driver to drop VLAN tags after they're used by the NIC to select an >> RX >> queue (since I need as many queues as cores). > So one other possibility that just occurred to me is that depending on > what it is you are trying to do you could just use one VLAN priority > tag and create a form of pseudo-RSS by just filtering on something > like the lowest 2 bits of the source and destination IP fields. That > would get you something like 16 queue RSS, but that assumes that your > traffic doesn't have to go to a specific queue and that you are just > wanting to break off your traffic into a separate group of queues and > not split traffic up by queue. That wouldn't fit my requirements. I want clients to be able to generate packets that are directed to a specific RX queue in a way that is *not* based on IPs or ports. Thanks, Lluis ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
