On Mon, Oct 16, 2017 at 3:53 AM, Lluís Vilanova <vilan...@ac.upc.edu> wrote: > Alexander Duyck writes: > >> On Fri, Oct 13, 2017 at 11:26 PM, Lluís Vilanova <vilan...@ac.upc.edu> wrote: >>> Hi! I'm not sure if this is the right venue for this question, so feel free >>> to >>> redirect me. >>> >>> I'm trying to put some "additional information" on client-generated packets >>> to >>> direct them to specific queues of a X540-AT2 nic (server running on Linux); >>> i.e., some type of client-directed queue balancing. >>> >>> So far, I've tried the following with no luck (any ideas?): >>> >>> * VLAN tag: >>> Assigning separate VLAN tags for each target queue has as a result that my >>> server receives no packets, since it's listening on the main NIC interface >>> which has no VLAN tags assigned. Is there a way to drop them once received, >>> so >>> that they'll only be used for "queue balancing"? >>> > >> So how many different queues do you need to divide your traffic into? >> If you can work with only 8 total classification types you might look >> at using the VLAN priority field and just use VLAN 0 for all your >> traffic. That would work around the redirecting to VLANs issue. > > Unfortunately I need 16 queues (one per core on the server host). > > >>> * TOS field (bits 8-15 of IPv4 header): >>> When I use ethtool to direct packets to each queue based on the TOS field, >>> the >>> nic seems to ignore it (TOS is always 0 on the rules): >>> >>> $ sudo ethtool -U em2 flow-type tcp4 tos 1 action 10 >>> Added rule with ID 2045 >>> $ ethtool --show-ntuple em2 >>> 16 RX rings available >>> Total 1 rules >>> Filter: 2045 >>> Rule Type: TCP over IPv4 >>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>> TOS: 0x0 mask: 0xff >>> Src port: 0 mask: 0xffff >>> Dest port: 0 mask: 0xffff >>> VLAN EtherType: 0x0 mask: 0xffff >>> VLAN: 0x0 mask: 0xffff >>> User-defined: 0x0 mask: 0xffffffffffffffff >>> Action: Direct to queue 10 >>> > >> Not surprising as the hardware doesn't support the TOS field for >> filtering. The fact that it was overlooked was a bug though. You could >> probably file a bug for that at e1000.sf.net > > True, I was looking at the wrong datasheet and thought TOS filtering was > available. > > >>> * user-def: >>> I've also tried with user-def to overcome the TOS field "problem", but it >>> seems I can only use that on the last two bytes: >>> >>> $ sudo ethtool -U em2 flow-type tcp4 user-def 2 action 10 >>> Added rule with ID 2045 >>> $ ethtool --show-ntuple em2 >>> 16 RX rings available >>> Total 1 rules >>> >>> Filter: 2045 >>> Rule Type: TCP over IPv4 >>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>> TOS: 0x0 mask: 0xff >>> Src port: 0 mask: 0xffff >>> Dest port: 0 mask: 0xffff >>> VLAN EtherType: 0x0 mask: 0xffff >>> VLAN: 0x0 mask: 0xffff >>> User-defined: 0x2 mask: 0xffffffffffffff00 >>> Action: Direct to queue 10 >>> > >> I'm not sure what you were hoping to do with the user-defined field. >> Last I knew the value was just used to indicate what VF that traffic >> was going to be routed to. > > I thought it would map into what the NIC's datasheet describes as: > > "Flexible 2-byte tuple anywhere in the first 64 bytes of the packet" > > So I was hoping to use it as a substitute to the TOS filter (which you told me > is not available on my specific NIC).
So the way the filter works is that it will only do a 2 byte tuple at a fixed offset based on the start of the Ethernet frame. This is somewhat limiting since a VLAN can shift that offset by 4 bytes so you cannot reliably use the offset value outside of the Ethernet header. That was one of the reasons why we had hard coded the offset to be the EtherType value that I believe is reported as VLAN EtherType since it will either be the EtherType for the inner frame, or the EtherType of the VLAN in the case of a VLAN being present. You could modify the driver to make the field fit what you want, but the limitation is that you would then have to make certain no VLAN tags are present on the frames, or encode the rule so that a certain VLAN tag must always be present when doing the filtering. >>> And when I try to match some other bytes, it's simply ignored (user-defined >>> is >>> always zero and the mask is full): >>> >>> $ sudo ethtool -U em2 flow-type tcp4 user-def 2 m 0xf0ffffffffffffff >>> action 10 >>> Added rule with ID 2045 >>> $ ethtool --show-ntuple em2 >>> 16 RX rings available >>> Total 1 rules >>> >>> Filter: 2045 >>> Rule Type: TCP over IPv4 >>> Src IP addr: 0.0.0.0 mask: 255.255.255.255 >>> Dest IP addr: 0.0.0.0 mask: 255.255.255.255 >>> TOS: 0x0 mask: 0xff >>> Src port: 0 mask: 0xffff >>> Dest port: 0 mask: 0xffff >>> VLAN EtherType: 0x0 mask: 0xffff >>> VLAN: 0x0 mask: 0xffff >>> User-defined: 0x0 mask: 0xffffffffffffffff >>> Action: Direct to queue 10 >>> >>> >>> Thanks! >>> Lluis > >> The filters for the ixgbe driver are pretty limited. As I recall you >> have an EtherType mask, VLAN tag, VLAN priority, source IP, >> destination IP, source port, destination port, and if it is a TCP over >> IPv4 or UDP over IPv4 rule. Like I said I think your best chance might >> be to look at encoding and filtering on VLAN priority but that only >> gives you about 8 queues you could target since that is composed of >> only 3 bits. > >> I hope that helps. > > Thanks a lot. If I can't use the "user-defined" filters I will see if I can > hack > into the driver to drop VLAN tags after they're used by the NIC to select an > RX > queue (since I need as many queues as cores). So one other possibility that just occurred to me is that depending on what it is you are trying to do you could just use one VLAN priority tag and create a form of pseudo-RSS by just filtering on something like the lowest 2 bits of the source and destination IP fields. That would get you something like 16 queue RSS, but that assumes that your traffic doesn't have to go to a specific queue and that you are just wanting to break off your traffic into a separate group of queues and not split traffic up by queue. - Alex ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
