Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Finn Christensen]

Shahaf,

No, I haven't looked into that issue, mainly because it seemed NIC-PMD related, 
rather than OVS.
I can try to make a simple comparison with a loopback test in OVS using a 
Napatech NIC and with/without HW acceleration.
Will get back to you on my findings.

Finn

From: Shahaf Shuler 
Sent: 22. juni 2018 09:07
To: Finn Christensen ; Koujalagi, MalleshX 
; y...@fridaylinux.org
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 
; Olga Shern 
Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Finn,

It is great that we are aligned w/ the finding.

I agree w/ your approach to take the relevant fields on the tcp_parse and limit 
the offload rules to hw based on validated action.

There was one more issue from flavio about the single core performance w/ HWOL.
He saw performance drop when enabling HWOL in a simple phy2phy loopback test.
I wasn't able to reproduce it, in fact saw big improvement.
Have you tried it? Would really like to address this issue before the v11.

From: Finn Christensen mailto:f...@napatech.com>>
Sent: Thursday, June 21, 2018 10:23:22 PM
To: Shahaf Shuler; Koujalagi, MalleshX; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A; Tsai, 
James; Olga Shern
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow


Hi Shahaf,



These are exactly the same bugs I found today.

I added the calculation of the offset l3_ofs and l4_ofs into the 
parse_tcp_flags() and here these can be calculated more or less without 
additional performance penalty.

I also made it not use the MARK in emc_processing if recirc depth > 0. Then 
VxLan with partial hw offload worked. However, I think the MARK also could be 
cleared in the tunnel pop function, before recirculation. To use md_is_valid 
flag is a bit more confusing to me.



I very much agree with you in your concerns, however, I do believe we must 
bypass the miniflow_extract, because otherwise nothing is gained. Then only 
full offload makes sense (or any other partly full offload, not defined yet).



To make sure we do not have such an issue again, which we have not thought of, 
we could change the flow offload check, so that we also check on the flow 
action list, before offloading into HW, and only accept the actions we have 
tested and knows work.

Then extend while we continue improve partial hw offload to cover more and more 
flows. To me it would make sense, also when extending to full offload, where 
all actions will have to be validated and accepted anyway.



Regarding inner match, I think it may be better handled by offloading 
VTEP/tunnel encap/decap in HW and then also have the possibility to match on 
inner packet while still in HW. But that's just my thoughts.



I'm all for starting limited, and then improve later.



/Finn



From: Shahaf Shuler mailto:shah...@mellanox.com>>
Sent: 21. juni 2018 19:26
To: Finn Christensen mailto:f...@napatech.com>>; Koujalagi, 
MalleshX mailto:malleshx.koujal...@intel.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>; Olga Shern 
mailto:ol...@mellanox.com>>
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow



Hi Finn,



Finally I was able to reproduce the error behavior w/ VXLAN traffic.



I found 2 issues related to the design being made:

  1.  The VXLAN decap requires more fields apart from the TCP header like the 
l3_ofs and l4_ofs which are missing because of bypassing the miniflow extract.
  2.  Tunnel packets are being recirculated after the outer headers decap 
(dp_netdev_recirculate). On the second processing stage the packet still 
carries the flow mark therefore the action for the inner headers Is also 
pop_tunnel.



I haven't established yet the final way to solve those issues. My current 
thoughts:

  1.  Even though attractive we cannot bypass the miniflow extract. VXLAN is 
just one case. Probably other cases will need different fields from the flow.
  2.  Use the md_is_valid flag on dp_netdev_input__ to recognize the non-first 
round and to skip the mark to flow search. I don't really like this approach 
because

 *   It is a bit hackish to relay on the md_is_valid. Alternatively we can 
clear the mbuf mark flag but this basically the same.
 *   w/ tunnel packets the massive amount of flow rules is in the inner 
headers. The outer doesn't change much. w/ the current design it means we 
offload the less consuming part.

Having said that,  the above approach looks the fastest w/o changing the 
current design. We can further improve this part later.



Be happy to hear more suggestions.





From: Finn Christensen [mailto:f...@napatech.com]
Sent: Thursday, June 21, 2018 5:46 PM
T

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Finn,

It is great that we are aligned w/ the finding.

I agree w/ your approach to take the relevant fields on the tcp_parse and limit 
the offload rules to hw based on validated action.

There was one more issue from flavio about the single core performance w/ HWOL.
He saw performance drop when enabling HWOL in a simple phy2phy loopback test.
I wasn’t able to reproduce it, in fact saw big improvement.
Have you tried it? Would really like to address this issue before the v11.

From: Finn Christensen 
Sent: Thursday, June 21, 2018 10:23:22 PM
To: Shahaf Shuler; Koujalagi, MalleshX; y...@fridaylinux.org
Cc: d...@openvswitch.org; Ergin, Mesut A; Tsai, James; Olga Shern
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow


Hi Shahaf,



These are exactly the same bugs I found today.

I added the calculation of the offset l3_ofs and l4_ofs into the 
parse_tcp_flags() and here these can be calculated more or less without 
additional performance penalty.

I also made it not use the MARK in emc_processing if recirc depth > 0. Then 
VxLan with partial hw offload worked. However, I think the MARK also could be 
cleared in the tunnel pop function, before recirculation. To use md_is_valid 
flag is a bit more confusing to me.



I very much agree with you in your concerns, however, I do believe we must 
bypass the miniflow_extract, because otherwise nothing is gained. Then only 
full offload makes sense (or any other partly full offload, not defined yet).



To make sure we do not have such an issue again, which we have not thought of, 
we could change the flow offload check, so that we also check on the flow 
action list, before offloading into HW, and only accept the actions we have 
tested and knows work.

Then extend while we continue improve partial hw offload to cover more and more 
flows. To me it would make sense, also when extending to full offload, where 
all actions will have to be validated and accepted anyway.



Regarding inner match, I think it may be better handled by offloading 
VTEP/tunnel encap/decap in HW and then also have the possibility to match on 
inner packet while still in HW. But that’s just my thoughts.



I’m all for starting limited, and then improve later.



/Finn



From: Shahaf Shuler 
Sent: 21. juni 2018 19:26
To: Finn Christensen ; Koujalagi, MalleshX 
; y...@fridaylinux.org
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 
; Olga Shern 
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow



Hi Finn,



Finally I was able to reproduce the error behavior w/ VXLAN traffic.



I found 2 issues related to the design being made:

  1.  The VXLAN decap requires more fields apart from the TCP header like the 
l3_ofs and l4_ofs which are missing because of bypassing the miniflow extract.
  2.  Tunnel packets are being recirculated after the outer headers decap 
(dp_netdev_recirculate). On the second processing stage the packet still 
carries the flow mark therefore the action for the inner headers Is also 
pop_tunnel.



I haven’t established yet the final way to solve those issues. My current 
thoughts:

  1.  Even though attractive we cannot bypass the miniflow extract. VXLAN is 
just one case. Probably other cases will need different fields from the flow.
  2.  Use the md_is_valid flag on dp_netdev_input__ to recognize the non-first 
round and to skip the mark to flow search. I don’t really like this approach 
because

 *   It is a bit hackish to relay on the md_is_valid. Alternatively we can 
clear the mbuf mark flag but this basically the same.
 *   w/ tunnel packets the massive amount of flow rules is in the inner 
headers. The outer doesn’t change much. w/ the current design it means we 
offload the less consuming part.

Having said that,  the above approach looks the fastest w/o changing the 
current design. We can further improve this part later.



Be happy to hear more suggestions.





From: Finn Christensen [mailto:f...@napatech.com]
Sent: Thursday, June 21, 2018 5:46 PM
To: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>; Shahaf 
Shuler mailto:shah...@mellanox.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow



Hi Mallesh and Shahaf,



I have tried to reproduce the issue Mallesh is reporting, using a Napatech NIC. 
The result is that I’m able to reproduce the error and are not getting the 
decap functionality to work when using VxLan tunneling together with partial 
hw-offload. Mainly the VxLAN POP is not performed in OVS in my setup.

The VxLan setup I’m using is straight forward with 2 hosts and br-int and 
br-phy bridges, like the typical examples.



I don’t know how far you have got into debugging thi

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Finn Christensen]

Hi Shahaf,

These are exactly the same bugs I found today.
I added the calculation of the offset l3_ofs and l4_ofs into the 
parse_tcp_flags() and here these can be calculated more or less without 
additional performance penalty.
I also made it not use the MARK in emc_processing if recirc depth > 0. Then 
VxLan with partial hw offload worked. However, I think the MARK also could be 
cleared in the tunnel pop function, before recirculation. To use md_is_valid 
flag is a bit more confusing to me.

I very much agree with you in your concerns, however, I do believe we must 
bypass the miniflow_extract, because otherwise nothing is gained. Then only 
full offload makes sense (or any other partly full offload, not defined yet).

To make sure we do not have such an issue again, which we have not thought of, 
we could change the flow offload check, so that we also check on the flow 
action list, before offloading into HW, and only accept the actions we have 
tested and knows work.
Then extend while we continue improve partial hw offload to cover more and more 
flows. To me it would make sense, also when extending to full offload, where 
all actions will have to be validated and accepted anyway.

Regarding inner match, I think it may be better handled by offloading 
VTEP/tunnel encap/decap in HW and then also have the possibility to match on 
inner packet while still in HW. But that's just my thoughts.

I'm all for starting limited, and then improve later.

/Finn

From: Shahaf Shuler 
Sent: 21. juni 2018 19:26
To: Finn Christensen ; Koujalagi, MalleshX 
; y...@fridaylinux.org
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 
; Olga Shern 
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Finn,

Finally I was able to reproduce the error behavior w/ VXLAN traffic.

I found 2 issues related to the design being made:

  1.  The VXLAN decap requires more fields apart from the TCP header like the 
l3_ofs and l4_ofs which are missing because of bypassing the miniflow extract.
  2.  Tunnel packets are being recirculated after the outer headers decap 
(dp_netdev_recirculate). On the second processing stage the packet still 
carries the flow mark therefore the action for the inner headers Is also 
pop_tunnel.

I haven't established yet the final way to solve those issues. My current 
thoughts:

  1.  Even though attractive we cannot bypass the miniflow extract. VXLAN is 
just one case. Probably other cases will need different fields from the flow.
  2.  Use the md_is_valid flag on dp_netdev_input__ to recognize the non-first 
round and to skip the mark to flow search. I don't really like this approach 
because

 *   It is a bit hackish to relay on the md_is_valid. Alternatively we can 
clear the mbuf mark flag but this basically the same.
 *   w/ tunnel packets the massive amount of flow rules is in the inner 
headers. The outer doesn't change much. w/ the current design it means we 
offload the less consuming part.

Having said that,  the above approach looks the fastest w/o changing the 
current design. We can further improve this part later.

Be happy to hear more suggestions.


From: Finn Christensen [mailto:f...@napatech.com]
Sent: Thursday, June 21, 2018 5:46 PM
To: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>; Shahaf 
Shuler mailto:shah...@mellanox.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Mallesh and Shahaf,

I have tried to reproduce the issue Mallesh is reporting, using a Napatech NIC. 
The result is that I'm able to reproduce the error and are not getting the 
decap functionality to work when using VxLan tunneling together with partial 
hw-offload. Mainly the VxLAN POP is not performed in OVS in my setup.
The VxLan setup I'm using is straight forward with 2 hosts and br-int and 
br-phy bridges, like the typical examples.

I don't know how far you have got into debugging this, but what I se is that 
the vxlan pop functionality needs packet information originally gathered in the 
miniflow extract function, which is omitted with hw-offload. Therefore, it does 
not work.

I'm still investigating how this may be solved, but I wanted to hear where you 
are in debugging this issue, and if you already have got to a solution.

Regards,
Finn

From: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>
Sent: 20. juni 2018 00:59
To: Shahaf Shuler mailto:shah...@mellanox.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>; Finn Christensen 
mailto:f...@napatech.com>>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] [PAT

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Hi Finn,

Finally I was able to reproduce the error behavior w/ VXLAN traffic.

I found 2 issues related to the design being made:

  1.  The VXLAN decap requires more fields apart from the TCP header like the 
l3_ofs and l4_ofs which are missing because of bypassing the miniflow extract.
  2.  Tunnel packets are being recirculated after the outer headers decap 
(dp_netdev_recirculate). On the second processing stage the packet still 
carries the flow mark therefore the action for the inner headers Is also 
pop_tunnel.

I haven't established yet the final way to solve those issues. My current 
thoughts:

  1.  Even though attractive we cannot bypass the miniflow extract. VXLAN is 
just one case. Probably other cases will need different fields from the flow.
  2.  Use the md_is_valid flag on dp_netdev_input__ to recognize the non-first 
round and to skip the mark to flow search. I don't really like this approach 
because
 *   It is a bit hackish to relay on the md_is_valid. Alternatively we can 
clear the mbuf mark flag but this basically the same.
 *   w/ tunnel packets the massive amount of flow rules is in the inner 
headers. The outer doesn't change much. w/ the current design it means we 
offload the less consuming part.

Having said that,  the above approach looks the fastest w/o changing the 
current design. We can further improve this part later.

Be happy to hear more suggestions.


From: Finn Christensen [mailto:f...@napatech.com]
Sent: Thursday, June 21, 2018 5:46 PM
To: Koujalagi, MalleshX ; Shahaf Shuler 
; y...@fridaylinux.org
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 

Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Mallesh and Shahaf,

I have tried to reproduce the issue Mallesh is reporting, using a Napatech NIC. 
The result is that I'm able to reproduce the error and are not getting the 
decap functionality to work when using VxLan tunneling together with partial 
hw-offload. Mainly the VxLAN POP is not performed in OVS in my setup.
The VxLan setup I'm using is straight forward with 2 hosts and br-int and 
br-phy bridges, like the typical examples.

I don't know how far you have got into debugging this, but what I se is that 
the vxlan pop functionality needs packet information originally gathered in the 
miniflow extract function, which is omitted with hw-offload. Therefore, it does 
not work.

I'm still investigating how this may be solved, but I wanted to hear where you 
are in debugging this issue, and if you already have got to a solution.

Regards,
Finn

From: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>
Sent: 20. juni 2018 00:59
To: Shahaf Shuler mailto:shah...@mellanox.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>; Finn Christensen 
mailto:f...@napatech.com>>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Shahaf,

Thanks for setting up VXLAN env. and trying to reproduce, appreciate!!

As you suggested, I moved to DPDK17.11.3 stable and OvS 2.9.0, still observed 
same issue.

An attached Vxlan_diagram.jpg, to get clear idea, we are seeing issue while 
traffic injected from Traffic Generator-->HOST B[eth1-->Vxlan-->eth0]-->Vxlan 
Tunnel Packet-->HOST A[eth0-->Vxlan-->eth1]-->Traffic Generator direction,  at 
Host A (eth0-->Vxlan-->eth1), we see Vxlan DECAP issue in case of HW-offload 
enabled. Vxlan Tunnel packets are not DECAP from eth0 to eth1, not observed any 
packet receiving @ eth1.

We injected random destination ip address(196.0.0.(0/1)),

  1.   Find tcpdump @ Host A (eth0).
07:24:12.013848 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.46816 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.1.1024: UDP, length 18
07:24:12.013860 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.50892 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.0.1024: UDP, length 18


  1.  Ovs logs.
  2.  Sure, further we will debug this issue using Skype/webex session. Please 
let me know good time.


Best regards
-/Mallesh


From: Shahaf Shuler [mailto:shah...@mellanox.com]
Sent: Monday, June 18, 2018 4:05 AM
To: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>; 
f...@napatech.com<mailto:f...@napatech.com>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] 

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Finn Christensen]

Hi Mallesh and Shahaf,

I have tried to reproduce the issue Mallesh is reporting, using a Napatech NIC. 
The result is that I'm able to reproduce the error and are not getting the 
decap functionality to work when using VxLan tunneling together with partial 
hw-offload. Mainly the VxLAN POP is not performed in OVS in my setup.
The VxLan setup I'm using is straight forward with 2 hosts and br-int and 
br-phy bridges, like the typical examples.

I don't know how far you have got into debugging this, but what I se is that 
the vxlan pop functionality needs packet information originally gathered in the 
miniflow extract function, which is omitted with hw-offload. Therefore, it does 
not work.

I'm still investigating how this may be solved, but I wanted to hear where you 
are in debugging this issue, and if you already have got to a solution.

Regards,
Finn

From: Koujalagi, MalleshX 
Sent: 20. juni 2018 00:59
To: Shahaf Shuler ; y...@fridaylinux.org; Finn 
Christensen 
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 

Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Shahaf,

Thanks for setting up VXLAN env. and trying to reproduce, appreciate!!

As you suggested, I moved to DPDK17.11.3 stable and OvS 2.9.0, still observed 
same issue.

An attached Vxlan_diagram.jpg, to get clear idea, we are seeing issue while 
traffic injected from Traffic Generator-->HOST B[eth1-->Vxlan-->eth0]-->Vxlan 
Tunnel Packet-->HOST A[eth0-->Vxlan-->eth1]-->Traffic Generator direction,  at 
Host A (eth0-->Vxlan-->eth1), we see Vxlan DECAP issue in case of HW-offload 
enabled. Vxlan Tunnel packets are not DECAP from eth0 to eth1, not observed any 
packet receiving @ eth1.

We injected random destination ip address(196.0.0.(0/1)),

  1.   Find tcpdump @ Host A (eth0).
07:24:12.013848 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.46816 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.1.1024: UDP, length 18
07:24:12.013860 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.50892 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.0.1024: UDP, length 18


  1.  Ovs logs.
  2.  Sure, further we will debug this issue using Skype/webex session. Please 
let me know good time.


Best regards
-/Mallesh


From: Shahaf Shuler [mailto:shah...@mellanox.com]
Sent: Monday, June 18, 2018 4:05 AM
To: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>; 
y...@fridaylinux.org<mailto:y...@fridaylinux.org>; 
f...@napatech.com<mailto:f...@napatech.com>
Cc: d...@openvswitch.org<mailto:d...@openvswitch.org>; Ergin, Mesut A 
mailto:mesut.a.er...@intel.com>>; Tsai, James 
mailto:james.t...@intel.com>>
Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Mallesh,

I was finally able to setup the vxlan testing w/ OVS. Instead of using OVS on 
both sides I used vxlan i/f to inject the traffic on one host and run ovs with 
vxlan tunnel configuration you specified on the other.

I was not able to reproduce your case. Actually I see the rules are created 
successfully (see attached ovs log "vxlan_tep")

Few observations:

  1.  The rule created for the VXLAN is for the outer pattern up to the UDP, 
hence supported by the device

 *   2018-06-18T08:11:17.466Z|00057|dpif_netdev(pmd53)|DBG|flow match: 
recirc_id=0,eth,udp,vlan_tci=0x,dl_src=e4:1d:2d:

ca:ca:6a,dl_dst=e4:1d:2d:a3:aa:74,nw_dst=2.2.2.1,nw_frag=no,tp_dst=4789

  1.  I do see segfault on mlx5 PMD on 17.11.0 and this was resolved on 17.11.3 
stable.
  2.  I used MLNX_OFED_LINUX-4.3-2.0.1.0, however I don't expect it to cause 
any diff

The packets being sent to the ovs w/ vxlan tunnel logic are[1]. tcpdump after 
the ovs logic is [2], the packet outer headers were removed as expected.

On top of all, I tried to force the validate function to fail. Indeed the flow 
was not created but the decap functionality still happens (see attached ovs log 
"vxlan_tep_force_err")

Can you please:

  1.  Provide me the type of packets you inject and don't see being decap.
  2.  Try w/ 17.11.3 stable to avoid any issues from the DPDK side
  3.  If you still see the issue can we set webex/skype meeting for joint 
debug, currently I don't see the error you reported on.


[1]
###[ Ethernet ]###
  dst= e4:1d:2d:a3:aa:74
  src= e4:1d:2d:ca:ca:6a
  type= IPv4
###[ IP ]###
 version= 4
 ihl= None
 tos= 0x0
 len= None
 id= 1
 flags=
 frag= 0
 ttl= 64
 proto= udp
 chksum= None
 src= 2.2.2.2
 dst= 2.2.2.1
 \options\
###[ UDP ]###
sport= 34550
dport= 4789
len= None
chksum= None
###[ VXLAN

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Koujalagi, MalleshX]

Hi Shahaf,

Thanks for setting up VXLAN env. and trying to reproduce, appreciate!!

As you suggested, I moved to DPDK17.11.3 stable and OvS 2.9.0, still observed 
same issue.

An attached Vxlan_diagram.jpg, to get clear idea, we are seeing issue while 
traffic injected from Traffic Generator-->HOST B[eth1-->Vxlan-->eth0]-->Vxlan 
Tunnel Packet-->HOST A[eth0-->Vxlan-->eth1]-->Traffic Generator direction,  at 
Host A (eth0-->Vxlan-->eth1), we see Vxlan DECAP issue in case of HW-offload 
enabled. Vxlan Tunnel packets are not DECAP from eth0 to eth1, not observed any 
packet receiving @ eth1.

We injected random destination ip address(196.0.0.(0/1)),

1.Find tcpdump @ Host A (eth0).
07:24:12.013848 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.46816 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.1.1024: UDP, length 18
07:24:12.013860 68:05:ca:33:ff:99 > 68:05:ca:33:fe:f9, ethertype IPv4 (0x0800), 
length 110: 172.168.1.2.50892 > 172.168.1.1.4789: VXLAN, flags [I] (0x08), vni 0
00:10:94:00:00:12 > 00:00:01:00:00:01, ethertype IPv4 (0x0800), length 60: 
197.0.0.0.1024 > 196.0.0.0.1024: UDP, length 18


2.   Ovs logs.

3.   Sure, further we will debug this issue using Skype/webex session. 
Please let me know good time.


Best regards
-/Mallesh


From: Shahaf Shuler [mailto:shah...@mellanox.com]
Sent: Monday, June 18, 2018 4:05 AM
To: Koujalagi, MalleshX ; y...@fridaylinux.org; 
f...@napatech.com
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 

Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Mallesh,

I was finally able to setup the vxlan testing w/ OVS. Instead of using OVS on 
both sides I used vxlan i/f to inject the traffic on one host and run ovs with 
vxlan tunnel configuration you specified on the other.

I was not able to reproduce your case. Actually I see the rules are created 
successfully (see attached ovs log "vxlan_tep")

Few observations:

1.   The rule created for the VXLAN is for the outer pattern up to the UDP, 
hence supported by the device

a.   2018-06-18T08:11:17.466Z|00057|dpif_netdev(pmd53)|DBG|flow match: 
recirc_id=0,eth,udp,vlan_tci=0x,dl_src=e4:1d:2d:

ca:ca:6a,dl_dst=e4:1d:2d:a3:aa:74,nw_dst=2.2.2.1,nw_frag=no,tp_dst=4789

2.   I do see segfault on mlx5 PMD on 17.11.0 and this was resolved on 
17.11.3 stable.

3.   I used MLNX_OFED_LINUX-4.3-2.0.1.0, however I don't expect it to cause 
any diff

The packets being sent to the ovs w/ vxlan tunnel logic are[1]. tcpdump after 
the ovs logic is [2], the packet outer headers were removed as expected.

On top of all, I tried to force the validate function to fail. Indeed the flow 
was not created but the decap functionality still happens (see attached ovs log 
"vxlan_tep_force_err")

Can you please:

1.   Provide me the type of packets you inject and don't see being decap.

2.   Try w/ 17.11.3 stable to avoid any issues from the DPDK side

3.   If you still see the issue can we set webex/skype meeting for joint 
debug, currently I don't see the error you reported on.


[1]
###[ Ethernet ]###
  dst= e4:1d:2d:a3:aa:74
  src= e4:1d:2d:ca:ca:6a
  type= IPv4
###[ IP ]###
 version= 4
 ihl= None
 tos= 0x0
 len= None
 id= 1
 flags=
 frag= 0
 ttl= 64
 proto= udp
 chksum= None
 src= 2.2.2.2
 dst= 2.2.2.1
 \options\
###[ UDP ]###
sport= 34550
dport= 4789
len= None
chksum= None
###[ VXLAN ]###
   flags= I
   reserved1= 0
   vni= 1000
   reserved2= 0x0
###[ Ethernet ]###
  dst= 00:00:5e:00:01:01
  src= 18:66:da:f5:37:64
  type= IPv4
###[ IP ]###
 version= 4
 ihl= None
 tos= 0x0
 len= None
 id= 1
 flags=
 frag= 0
 ttl= 64
 proto= tcp
 chksum= None
 src= 10.7.12.62
 dst= 1.1.1.1
 \options\
###[ TCP ]###
sport= ftp_data
dport= http
seq= 0
ack= 0
dataofs= None
reserved= 0
flags= S
window= 8192
chksum= None
urgptr= 0
options= {}

[2]
11:13:59.477667 18:66:da:f5:37:64 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), 
length 60: (tos 0x0, ttl 64, id 1, off
set 0, flags [none], proto TCP (6), length 40)
10.7.12.62.20 > 1.1.1.1.80: Flags [S], cksum 0x7738 (correct), seq 0, win 
8192, length 0

--Shahaf

From: Koujalagi, MalleshX 
mailto:malleshx.koujal...@intel.com>>
S

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Mallesh,

I was finally able to setup the vxlan testing w/ OVS. Instead of using OVS on 
both sides I used vxlan i/f to inject the traffic on one host and run ovs with 
vxlan tunnel configuration you specified on the other.

I was not able to reproduce your case. Actually I see the rules are created 
successfully (see attached ovs log "vxlan_tep")

Few observations:

1.   The rule created for the VXLAN is for the outer pattern up to the UDP, 
hence supported by the device

a.   2018-06-18T08:11:17.466Z|00057|dpif_netdev(pmd53)|DBG|flow match: 
recirc_id=0,eth,udp,vlan_tci=0x,dl_src=e4:1d:2d:

ca:ca:6a,dl_dst=e4:1d:2d:a3:aa:74,nw_dst=2.2.2.1,nw_frag=no,tp_dst=4789

2.   I do see segfault on mlx5 PMD on 17.11.0 and this was resolved on 
17.11.3 stable.

3.   I used MLNX_OFED_LINUX-4.3-2.0.1.0, however I don't expect it to cause 
any diff

The packets being sent to the ovs w/ vxlan tunnel logic are[1]. tcpdump after 
the ovs logic is [2], the packet outer headers were removed as expected.

On top of all, I tried to force the validate function to fail. Indeed the flow 
was not created but the decap functionality still happens (see attached ovs log 
"vxlan_tep_force_err")

Can you please:

1.   Provide me the type of packets you inject and don't see being decap.

2.   Try w/ 17.11.3 stable to avoid any issues from the DPDK side

3.   If you still see the issue can we set webex/skype meeting for joint 
debug, currently I don't see the error you reported on.


[1]
###[ Ethernet ]###
  dst= e4:1d:2d:a3:aa:74
  src= e4:1d:2d:ca:ca:6a
  type= IPv4
###[ IP ]###
 version= 4
 ihl= None
 tos= 0x0
 len= None
 id= 1
 flags=
 frag= 0
 ttl= 64
 proto= udp
 chksum= None
 src= 2.2.2.2
 dst= 2.2.2.1
 \options\
###[ UDP ]###
sport= 34550
dport= 4789
len= None
chksum= None
###[ VXLAN ]###
   flags= I
   reserved1= 0
   vni= 1000
   reserved2= 0x0
###[ Ethernet ]###
  dst= 00:00:5e:00:01:01
  src= 18:66:da:f5:37:64
  type= IPv4
###[ IP ]###
 version= 4
 ihl= None
 tos= 0x0
 len= None
 id= 1
 flags=
 frag= 0
 ttl= 64
 proto= tcp
 chksum= None
 src= 10.7.12.62
 dst= 1.1.1.1
 \options\
###[ TCP ]###
sport= ftp_data
dport= http
seq= 0
ack= 0
dataofs= None
reserved= 0
flags= S
window= 8192
chksum= None
urgptr= 0
options= {}

[2]
11:13:59.477667 18:66:da:f5:37:64 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), 
length 60: (tos 0x0, ttl 64, id 1, off
set 0, flags [none], proto TCP (6), length 40)
10.7.12.62.20 > 1.1.1.1.80: Flags [S], cksum 0x7738 (correct), seq 0, win 
8192, length 0

--Shahaf

From: Koujalagi, MalleshX 
Sent: Friday, June 15, 2018 9:29 PM
To: Shahaf Shuler ; y...@fridaylinux.org; 
f...@napatech.com
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 

Subject: RE: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Shahaf,

Thanks for pointing NIC/protocol support.

Find inline comments:

>>When you say DECAP functionality is broken you mean the flow is not actually 
>>inserted to the HW right?
 [Mallesh]: Yes, DECAP flows are not inserted to HW.

>>The datapath should still DECAP the flow correctly.
 [Mallesh]: Agree, However the datapath failed to DECAP.
 If VXLAN protocol support in netdev_dpdk_validate_flow failed (return -1) 
then, should be fall back normal or default behavior of datapath right?

Have you tried w/o VXLAN tunnel rules?
[Mallesh]: Yes, tried, but same behavior.

Best regards,
-/Mallesh
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Koujalagi, MalleshX]

Hi Shahaf,

Thanks for pointing NIC/protocol support.

Find inline comments:

>>When you say DECAP functionality is broken you mean the flow is not actually 
>>inserted to the HW right?
 [Mallesh]: Yes, DECAP flows are not inserted to HW.

>>The datapath should still DECAP the flow correctly.
 [Mallesh]: Agree, However the datapath failed to DECAP.
 If VXLAN protocol support in netdev_dpdk_validate_flow failed (return -1) 
then, should be fall back normal or default behavior of datapath right?

Have you tried w/o VXLAN tunnel rules?
[Mallesh]: Yes, tried, but same behavior.

Best regards,
-/Mallesh
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Hi MalleshX,

The error you get is because the specific flow you try to offload is not yet 
supported by this patchset.

This patchset only provide the first stage of partial offload in OVS-DPDK and 
supporting only few protocols as stated on the release notes[1]
You can check out all the unsupported options in the function 
netdev_dpdk_validate_flow.

When you say DECAP functionality is broken you mean the flow is not actually 
inserted to the HW right?
The datapath should still decap the flow correctly.

Have you tried w/o VXLAN tunnel rules?
Pay attention to the small fix I provided Flavio, I will send soon the v11 
including it.

[1]
+.. _dpdk-flow-hardware-offload:
+
+Flow Hardware Offload (Experimental)
+
+
+The flow hardware offload is disabled by default and can be enabled by::
+
+$ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
+
+So far only partial flow offload is implemented. Moreover, it only works
+with PMD drivers have the rte_flow action "MARK + RSS" support.
+
+The validated NICs are:
+
+- Mellanox (ConnectX-4, ConnectX-4 Lx, ConnectX-5)
+- Napatech (NT200B01)
+
+Supported protocols for hardware offload are:
+- L2: Ethernet, VLAN
+- L3: IPv4, IPv6
+- L4: TCP, UDP, SCTP, ICMP

--Shahaf

From: Koujalagi, MalleshX 
Sent: Thursday, June 14, 2018 4:35 AM
To: y...@fridaylinux.org; Shahaf Shuler ; 
f...@napatech.com
Cc: d...@openvswitch.org; Ergin, Mesut A ; Tsai, James 

Subject: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Hi Yuanhan/Finn/Shahaf,

I tried hw-offloading  patch on top OvS v2.9.0 and DPDK 17.11.0 with VXLAN 
encap/decap functionalities. VXLAN encap is working fine, however VXLAN decap 
functionality is broken, Please find HW/SW conf and Setup as below:

[HW/SW configuration]:
CPU:  Intel(R) Xeon(R) Platinum 8180 28C @ 
2.50GHz
RAM: 128GB DDR4 2666
Linux Kernel:  4.4.0
Linux Dist.   Ubuntu 16.04.4 LTS
DPDK:   v17.11.0 (Std configuration)
OVS:  v2.9.0
Gcc:   5.4.0
NIC:   2 x MCX416A-CCAT- ConnectX-4 network 
interface card 100GbE dual-port QSFP28; (version: 4.3-1.0.1 firmware-version: 
12.22.1002 (MT_2150110033)
Traffic Type:   64-byte Ethernet Frames with UDP/IP 
payload, Uniform Random Flows Sweeping Destination IP address
NUMA/Core setting:  on Socket #1

[VXLAN Setup]: Following section 8.5.1 VXLAN Test Methodology
https://download.01.org/packet-processing/ONPS2.0/Intel_ONP_Release_2.0_Performance_Test_Report_Rev1.0-1.pdf

[Logs]:
2018-06-13T16:02:51.712Z|00155|netdev_linux|INFO|ioctl(SIOCGIFINDEX) on 
vxlan_sys_4789 device failed: No such device
2018-06-13T16:02:51.723Z|00156|connmgr|INFO|br-int<->unix#2: 1 flow_mods in the 
last 0 s (1 adds)
2018-06-13T16:02:51.725Z|00157|connmgr|INFO|br-int<->unix#4: 1 flow_mods in the 
last 0 s (1 adds)
2018-06-13T16:03:00.354Z|1|netdev_linux(dp_netdev_flow_59)|INFO|ioctl(SIOCGIFINDEX)
 on vxlan_sys_4789 device failed: No such device
2018-06-13T16:03:00.354Z|2|netdev_tc_offloads(dp_netdev_flow_59)|ERR|flow_put:
 failed to get ifindex for vxlan_sys_4789: No such device
2018-06-13T16:08:40.767Z|3|netdev_dpdk(dp_netdev_flow_59)|ERR|cannot HW 
accelerate this flow due to unsupported protocols


After disabling HW-offload,  VXLAN ENCAP/DECAP functionalities are worked fine!!

Best regards
-/Mallesh
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Hi Flavio,

I repeatedly tried to reproduce the results you have but failed. In fact I am 
now see better results for the hw-offload (thanks to optimization I learned 
from your script). 

w/o hw-offload I get fwd rate of ~7.76Mpps 

pmd thread numa_id 1 core_id 1:  
packets received: 475469632  
packet recirculations: 0 
avg. datapath passes per packet: 1.00
emc hits: 475466456  
megaflow hits: 3072  
avg. subtable lookups per megaflow hit: 1.00 
miss with success upcall: 1  
miss with failed upcall: 71  
avg. packets per output batch: 20.75 
idle cycles: 0 (0.00%)   
processing cycles: 135318738225 (100.00%)
avg cycles per packet: 284.60 (135318738225/475469632)   
avg processing cycles per packet: 284.60 (135318738225/475469632)

w/ hw-offload I get fwd rate of ~12Mpps. 

pmd thread numa_id 1 core_id 1:  
packets received: 213505047  
packet recirculations: 0 
avg. datapath passes per packet: 1.00
emc hits: 213502401  
megaflow hits: 2554  
avg. subtable lookups per megaflow hit: 1.00 
miss with success upcall: 1  
miss with failed upcall: 71  
avg. packets per output batch: 20.09 
idle cycles: 0 (0.00%)   
processing cycles: 40049794229 (100.00%) 
avg cycles per packet: 187.58 (40049794229/213505047)
avg processing cycles per packet: 187.58 (40049794229/213505047)

around 71% improvement and ~100 cycles of the datapath were saved by the HW 
offload.
Moreover, this is the most simple configuration. With more flows we can assume 
the gain will be much higher. 

Am wondering if a third person can check it to understand which system among 
the two is configured badly. 
Ian/Finn - can you please have a try? With Flavio scripts is really simple. 
BTW - you can use the latest stable tree of 17.11 (what will be 17.11.3), it 
includes the custom fixes I gave Flavio. 

More below 

Thursday, June 7, 2018 10:15 PM, Flavio Leitner:
> Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> On Thu, Jun 07, 2018 at 07:24:25AM +, Shahaf Shuler wrote:
> > Thanks Flavio for the update,
> >
> > Wednesday, June 6, 2018 5:48 PM, Flavio Leitner:
> > > Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with
> > > rte_flow
> > >
> > >
> > > (Looks like Intel doesn't allow shell scripts, so if you need I will
> > > put somewhere else to download)
> >
> > If you can pass me the scripts for your run it will be great, please include
> also packet gen pattern. I will try to reproduce it in house.
> > Looks like the flow is inserted correctly now.
> >
> > From my testing using single loopback rule, single core, single queue and
> single flow injected I can see ~20% improvement with the HW offload (from
> 5.2Mpps -> 6.2 Mpps).
> 
> Looking forward to get the same over here.
> 
> I added the Xena script to create the traffic and the script that brings up 
> the
> environment here:
> https://emea01.safelinks.protection.outlook.com/?url=http:%2F%2Fpeople.
> redhat.com%2F~fleitner%2Fhwol%2F=02%7C01%7Cshahafs%40mellan
> ox.com%7C0249e6ca39384c11979508d5ccab0556%7Ca652971c7d2e4d9ba6a4
> d149256f461b%7C0%7C1%7C636639957263712271=fWCBNjBGTpP1DS
> Nk6vHSebgyIrbkc1mnmvm%2BiLfJqd4%3D=0

Unfortunately I don't have Xena traffic generator. I used SW based traffic 
generator w/ 128 IP rules in round robin to reach my results. 
Is it possible some packets are dropped due to integrity reasons (CRC/checksum) 
? 

Can you attach the "ovs-appctl dpif-netdev/pmd-stats-show" output of each case?


> 
> Below is the packet headers I am providing to the traffic generator which will
> pad to 64 bytes, and increase the address from 10.0.0.1 to 10.0.0.128:
> 
>   1 0.00   10.0.0.1  10.0.0.2  IPv4 34
> 
> Frame 1: 34 bytes on wire (272 bits), 34 bytes 

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Flavio Leitner]

On Tue, May 22, 2018 at 10:27:07AM -0300, Flavio Leitner wrote:
> On Mon, May 21, 2018 at 02:25:56PM +, Shahaf Shuler wrote:
> > Hi Flavio, thanks for reporting this issue.
> > 
> > Saturday, May 19, 2018 1:47 AM, Flavio Leitner:
> > > Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> > > 
> > > 
> > > 
> > > Hello,
> > > 
> > > I looked at the patchset (v9) and I found no obvious problems, but I miss
> > > some instrumentation to understand what is going on. For example, how
> > > many flows are offloaded, or how many per second, etc... We can definitely
> > > work on that as a follow up.
> > > 
> > > I have a MLX5 (16.20.1010) which is connected to a traffic generator.
> > > The results are unexpected and I don't know why yet. I will continue on it
> > > next week.
> > > 
> > > The flow is pretty simple, just echo the packet back:
> > > 
> > >ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port
> > > 
> > > This is the result without HW offloading enabled:
> > > Partial:  14619675.00 pps  7485273570.00 bps
> > > Partial:  14652351.00 pps  7502003940.00 bps
> > > Partial:  14655548.00 pps  7503640570.00 bps
> > > Partial:  14679556.00 pps  7515932630.00 bps
> > > Partial:  14681188.00 pps  7516768670.00 bps
> > > Partial:  14597427.00 pps  7473882390.00 bps
> > > Partial:  14712617.00 pps  7532860090.00 bps
> > > 
> > > pmd thread numa_id 0 core_id 2:
> > > packets received: 53859055
> > > packet recirculations: 0
> > > avg. datapath passes per packet: 1.00
> > > emc hits: 53859055
> > > megaflow hits: 0
> > > avg. subtable lookups per megaflow hit: 0.00
> > > miss with success upcall: 0
> > > miss with failed upcall: 0
> > > avg. packets per output batch: 28.20
> > > idle cycles: 0 (0.00%)
> > > processing cycles: 12499399115 (100.00%)
> > > avg cycles per packet: 232.08 (12499399115/53859055)
> > > avg processing cycles per packet: 232.08 (12499399115/53859055)
> > > 
> > > 
> > > Based on the stats, it seems 14.7Mpps is the maximum a core can do it.
> > > 
> > > This is the result with HW Offloading enabled:
> > > 
> > > Partial:  10713500.00 pps  5485312330.00 bps
> > > Partial:  10672185.00 pps  5464158240.00 bps
> > > Partial:  10747756.00 pps  5502850960.00 bps
> > > Partial:  10713404.00 pps  5485267400.00 bps
> > > 
> > > 
> > > pmd thread numa_id 0 core_id 2:
> > > packets received: 25902718
> > > packet recirculations: 0
> > > avg. datapath passes per packet: 1.00
> > > emc hits: 25902697
> > > megaflow hits: 0
> > > avg. subtable lookups per megaflow hit: 0.00
> > > miss with success upcall: 0
> > > miss with failed upcall: 0
> > > avg. packets per output batch: 28.11
> > > idle cycles: 0 (0.00%)
> > > processing cycles: 12138284463 (100.00%)
> > > avg cycles per packet: 468.61 (12138284463/25902718)
> > > avg processing cycles per packet: 468.61 (12138284463/25902718)
> > > 
> > > 2018-05-
> > > 18T22:34:57.865Z|1|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > > id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b40 was not found
> > > 
> > > 2018-05-
> > > 18T22:35:02.920Z|2|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > > id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a4584c was not found
> > > 
> > 
> > Up till now makes sense. The first packet should trigger the flow insertion 
> > to HW.
> > 
> > > 2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident set
> > > size after 10.4 seconds
> > > 
> > > 2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
> > > revalidators:1 rules:5 udpif keys:2
> > > 
> > > 2018-05-
> > > 18T22:35:21.910Z|3|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > > id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc1893e was not found
> > > 
> > > 2018-05-18T22:35:21.924Z|4|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
> > > flow creat error: 2 : message : flow rule creation failure
> > 
> > This error is unexpected. 
> > I was able to reproduce the issue w

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Flavio Leitner]

On Mon, May 21, 2018 at 02:25:56PM +, Shahaf Shuler wrote:
> Hi Flavio, thanks for reporting this issue.
> 
> Saturday, May 19, 2018 1:47 AM, Flavio Leitner:
> > Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> > 
> > 
> > 
> > Hello,
> > 
> > I looked at the patchset (v9) and I found no obvious problems, but I miss
> > some instrumentation to understand what is going on. For example, how
> > many flows are offloaded, or how many per second, etc... We can definitely
> > work on that as a follow up.
> > 
> > I have a MLX5 (16.20.1010) which is connected to a traffic generator.
> > The results are unexpected and I don't know why yet. I will continue on it
> > next week.
> > 
> > The flow is pretty simple, just echo the packet back:
> > 
> >ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port
> > 
> > This is the result without HW offloading enabled:
> > Partial:  14619675.00 pps  7485273570.00 bps
> > Partial:  14652351.00 pps  7502003940.00 bps
> > Partial:  14655548.00 pps  7503640570.00 bps
> > Partial:  14679556.00 pps  7515932630.00 bps
> > Partial:  14681188.00 pps  7516768670.00 bps
> > Partial:  14597427.00 pps  7473882390.00 bps
> > Partial:  14712617.00 pps  7532860090.00 bps
> > 
> > pmd thread numa_id 0 core_id 2:
> > packets received: 53859055
> > packet recirculations: 0
> > avg. datapath passes per packet: 1.00
> > emc hits: 53859055
> > megaflow hits: 0
> > avg. subtable lookups per megaflow hit: 0.00
> > miss with success upcall: 0
> > miss with failed upcall: 0
> > avg. packets per output batch: 28.20
> > idle cycles: 0 (0.00%)
> > processing cycles: 12499399115 (100.00%)
> > avg cycles per packet: 232.08 (12499399115/53859055)
> > avg processing cycles per packet: 232.08 (12499399115/53859055)
> > 
> > 
> > Based on the stats, it seems 14.7Mpps is the maximum a core can do it.
> > 
> > This is the result with HW Offloading enabled:
> > 
> > Partial:  10713500.00 pps  5485312330.00 bps
> > Partial:  10672185.00 pps  5464158240.00 bps
> > Partial:  10747756.00 pps  5502850960.00 bps
> > Partial:  10713404.00 pps  5485267400.00 bps
> > 
> > 
> > pmd thread numa_id 0 core_id 2:
> > packets received: 25902718
> > packet recirculations: 0
> > avg. datapath passes per packet: 1.00
> > emc hits: 25902697
> > megaflow hits: 0
> > avg. subtable lookups per megaflow hit: 0.00
> > miss with success upcall: 0
> > miss with failed upcall: 0
> > avg. packets per output batch: 28.11
> > idle cycles: 0 (0.00%)
> > processing cycles: 12138284463 (100.00%)
> > avg cycles per packet: 468.61 (12138284463/25902718)
> > avg processing cycles per packet: 468.61 (12138284463/25902718)
> > 
> > 2018-05-
> > 18T22:34:57.865Z|1|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b40 was not found
> > 
> > 2018-05-
> > 18T22:35:02.920Z|2|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a4584c was not found
> > 
> 
> Up till now makes sense. The first packet should trigger the flow insertion 
> to HW.
> 
> > 2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident set
> > size after 10.4 seconds
> > 
> > 2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
> > revalidators:1 rules:5 udpif keys:2
> > 
> > 2018-05-
> > 18T22:35:21.910Z|3|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc1893e was not found
> > 
> > 2018-05-18T22:35:21.924Z|4|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
> > flow creat error: 2 : message : flow rule creation failure
> 
> This error is unexpected. 
> I was able to reproduce the issue with latest 17.11 stable. Looks like either 
> modify flow is broken on the DPDK side or OVS-DPDK issue which popped up due 
> to some commit on stable DPDK. 
> I will investigate more on it and provide a fix. 

Cool, thanks.


> For the sake of testing, can you try with 17.11 instead of stable? 
> With it I found it work correctly:

I will try in the next hours.


> 2018-05-21T14:20:32.954Z|00120|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.90 
> │
> 2018-05-21T14:20:33.906Z|00121|connmgr|INFO|br0<->unix#3: 1 flow_mods in the 
> last 0 s (1 adds)│
> 2018-05-21T14:20:33.923Z|2|dpif_netdev(dp_netdev_flow_4)|DBG|succeed to 
> modify netdev flow  
> 
> BTW - if you want to avoid the modify flow, try to insert the rule before you 
> start the traffic on the traffic gen. 
> The below output is the expected (in both LTS and 17.11 versions):

OK, thanks.

-- 
Flavio


___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Stokes, Ian]

> Hi Flavio, thanks for reporting this issue.
> 
> Saturday, May 19, 2018 1:47 AM, Flavio Leitner:
> > Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with
> > rte_flow
> >
> >
> >
> > Hello,
> >
> > I looked at the patchset (v9) and I found no obvious problems, but I
> > miss some instrumentation to understand what is going on. For example,
> > how many flows are offloaded, or how many per second, etc... We can
> > definitely work on that as a follow up.
> >
> > I have a MLX5 (16.20.1010) which is connected to a traffic generator.
> > The results are unexpected and I don't know why yet. I will continue
> > on it next week.
> >
> > The flow is pretty simple, just echo the packet back:
> >
> >ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port
> >
> > This is the result without HW offloading enabled:
> > Partial:  14619675.00 pps  7485273570.00 bps
> > Partial:  14652351.00 pps  7502003940.00 bps
> > Partial:  14655548.00 pps  7503640570.00 bps
> > Partial:  14679556.00 pps  7515932630.00 bps
> > Partial:  14681188.00 pps  7516768670.00 bps
> > Partial:  14597427.00 pps  7473882390.00 bps
> > Partial:  14712617.00 pps  7532860090.00 bps
> >
> > pmd thread numa_id 0 core_id 2:
> > packets received: 53859055
> > packet recirculations: 0
> > avg. datapath passes per packet: 1.00
> > emc hits: 53859055
> > megaflow hits: 0
> > avg. subtable lookups per megaflow hit: 0.00
> > miss with success upcall: 0
> > miss with failed upcall: 0
> > avg. packets per output batch: 28.20
> > idle cycles: 0 (0.00%)
> > processing cycles: 12499399115 (100.00%)
> > avg cycles per packet: 232.08 (12499399115/53859055)
> > avg processing cycles per packet: 232.08
> > (12499399115/53859055)
> >
> >
> > Based on the stats, it seems 14.7Mpps is the maximum a core can do it.
> >
> > This is the result with HW Offloading enabled:
> >
> > Partial:  10713500.00 pps  5485312330.00 bps
> > Partial:  10672185.00 pps  5464158240.00 bps
> > Partial:  10747756.00 pps  5502850960.00 bps
> > Partial:  10713404.00 pps  5485267400.00 bps
> >
> >
> > pmd thread numa_id 0 core_id 2:
> > packets received: 25902718
> > packet recirculations: 0
> > avg. datapath passes per packet: 1.00
> > emc hits: 25902697
> > megaflow hits: 0
> > avg. subtable lookups per megaflow hit: 0.00
> > miss with success upcall: 0
> > miss with failed upcall: 0
> > avg. packets per output batch: 28.11
> > idle cycles: 0 (0.00%)
> > processing cycles: 12138284463 (100.00%)
> > avg cycles per packet: 468.61 (12138284463/25902718)
> > avg processing cycles per packet: 468.61
> > (12138284463/25902718)
> >
> > 2018-05-
> > 18T22:34:57.865Z|1|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b40 was not found
> >
> > 2018-05-
> > 18T22:35:02.920Z|2|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a4584c was not found
> >
> 
> Up till now makes sense. The first packet should trigger the flow
> insertion to HW.
> 
> > 2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident set
> > size after 10.4 seconds
> >
> > 2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
> > revalidators:1 rules:5 udpif keys:2
> >
> > 2018-05-
> > 18T22:35:21.910Z|3|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> > id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc1893e was not found
> >
> > 2018-05-18T22:35:21.924Z|4|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
> > flow creat error: 2 : message : flow rule creation failure
> 
> This error is unexpected.
> I was able to reproduce the issue with latest 17.11 stable. Looks like
> either modify flow is broken on the DPDK side or OVS-DPDK issue which
> popped up due to some commit on stable DPDK.
> I will investigate more on it and provide a fix.

Currently we recommend DPDK 17.11.2 as it addresses a number of CVE fixes for 
vhost user. 

If the bug is specific to the DPDK side you could list it as a known issue 
under "Limitations' in Documentation/intro/install/dpdk.rst until it is fixed 
in a future release of 17.11.

Ian
> 
> For the sake of testing, can you try with 17.11 instead of stable?
> With it I found it work correctly:

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Hi Flavio, thanks for reporting this issue.

Saturday, May 19, 2018 1:47 AM, Flavio Leitner:
> Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> 
> 
> Hello,
> 
> I looked at the patchset (v9) and I found no obvious problems, but I miss
> some instrumentation to understand what is going on. For example, how
> many flows are offloaded, or how many per second, etc... We can definitely
> work on that as a follow up.
> 
> I have a MLX5 (16.20.1010) which is connected to a traffic generator.
> The results are unexpected and I don't know why yet. I will continue on it
> next week.
> 
> The flow is pretty simple, just echo the packet back:
> 
>ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port
> 
> This is the result without HW offloading enabled:
> Partial:  14619675.00 pps  7485273570.00 bps
> Partial:  14652351.00 pps  7502003940.00 bps
> Partial:  14655548.00 pps  7503640570.00 bps
> Partial:  14679556.00 pps  7515932630.00 bps
> Partial:  14681188.00 pps  7516768670.00 bps
> Partial:  14597427.00 pps  7473882390.00 bps
> Partial:  14712617.00 pps  7532860090.00 bps
> 
> pmd thread numa_id 0 core_id 2:
> packets received: 53859055
> packet recirculations: 0
> avg. datapath passes per packet: 1.00
> emc hits: 53859055
> megaflow hits: 0
> avg. subtable lookups per megaflow hit: 0.00
> miss with success upcall: 0
> miss with failed upcall: 0
> avg. packets per output batch: 28.20
> idle cycles: 0 (0.00%)
> processing cycles: 12499399115 (100.00%)
> avg cycles per packet: 232.08 (12499399115/53859055)
> avg processing cycles per packet: 232.08 (12499399115/53859055)
> 
> 
> Based on the stats, it seems 14.7Mpps is the maximum a core can do it.
> 
> This is the result with HW Offloading enabled:
> 
> Partial:  10713500.00 pps  5485312330.00 bps
> Partial:  10672185.00 pps  5464158240.00 bps
> Partial:  10747756.00 pps  5502850960.00 bps
> Partial:  10713404.00 pps  5485267400.00 bps
> 
> 
> pmd thread numa_id 0 core_id 2:
> packets received: 25902718
> packet recirculations: 0
> avg. datapath passes per packet: 1.00
> emc hits: 25902697
> megaflow hits: 0
> avg. subtable lookups per megaflow hit: 0.00
> miss with success upcall: 0
> miss with failed upcall: 0
> avg. packets per output batch: 28.11
> idle cycles: 0 (0.00%)
> processing cycles: 12138284463 (100.00%)
> avg cycles per packet: 468.61 (12138284463/25902718)
> avg processing cycles per packet: 468.61 (12138284463/25902718)
> 
> 2018-05-
> 18T22:34:57.865Z|1|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b40 was not found
> 
> 2018-05-
> 18T22:35:02.920Z|2|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a4584c was not found
> 

Up till now makes sense. The first packet should trigger the flow insertion to 
HW.

> 2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident set
> size after 10.4 seconds
> 
> 2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
> revalidators:1 rules:5 udpif keys:2
> 
> 2018-05-
> 18T22:35:21.910Z|3|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc1893e was not found
> 
> 2018-05-18T22:35:21.924Z|4|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
> flow creat error: 2 : message : flow rule creation failure

This error is unexpected. 
I was able to reproduce the issue with latest 17.11 stable. Looks like either 
modify flow is broken on the DPDK side or OVS-DPDK issue which popped up due to 
some commit on stable DPDK. 
I will investigate more on it and provide a fix. 

For the sake of testing, can you try with 17.11 instead of stable? 
With it I found it work correctly:

2018-05-21T14:20:32.954Z|00120|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.90   
  │
2018-05-21T14:20:33.906Z|00121|connmgr|INFO|br0<->unix#3: 1 flow_mods in the 
last 0 s (1 adds)│
2018-05-21T14:20:33.923Z|2|dpif_netdev(dp_netdev_flow_4)|DBG|succeed to 
modify netdev flow  

BTW - if you want to avoid the modify flow, try to insert the rule before you 
start the traffic on the traffic gen. 
The below output is the expected (in both LTS and 17.11 versions):


2018-05-21T13:19:18.055Z|00120|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.90   
  
2018-05-21T13:19:23.560Z|00121|memory|INFO|294204 kB peak resident set size 
after 10.4 seconds
2018-05-21T13:19:23.560Z|00122|memory

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Flavio Leitner]

Hello,

I looked at the patchset (v9) and I found no obvious problems, but I
miss some instrumentation to understand what is going on. For example,
how many flows are offloaded, or how many per second, etc... We can
definitely work on that as a follow up.

I have a MLX5 (16.20.1010) which is connected to a traffic generator.
The results are unexpected and I don't know why yet. I will continue
on it next week.

The flow is pretty simple, just echo the packet back:

   ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port

This is the result without HW offloading enabled:
Partial:  14619675.00 pps  7485273570.00 bps
Partial:  14652351.00 pps  7502003940.00 bps
Partial:  14655548.00 pps  7503640570.00 bps
Partial:  14679556.00 pps  7515932630.00 bps
Partial:  14681188.00 pps  7516768670.00 bps
Partial:  14597427.00 pps  7473882390.00 bps
Partial:  14712617.00 pps  7532860090.00 bps

pmd thread numa_id 0 core_id 2:
packets received: 53859055
packet recirculations: 0
avg. datapath passes per packet: 1.00
emc hits: 53859055
megaflow hits: 0
avg. subtable lookups per megaflow hit: 0.00
miss with success upcall: 0
miss with failed upcall: 0
avg. packets per output batch: 28.20
idle cycles: 0 (0.00%)
processing cycles: 12499399115 (100.00%)
avg cycles per packet: 232.08 (12499399115/53859055)
avg processing cycles per packet: 232.08 (12499399115/53859055)


Based on the stats, it seems 14.7Mpps is the maximum a core can do it.

This is the result with HW Offloading enabled:

Partial:  10713500.00 pps  5485312330.00 bps
Partial:  10672185.00 pps  5464158240.00 bps
Partial:  10747756.00 pps  5502850960.00 bps
Partial:  10713404.00 pps  5485267400.00 bps


pmd thread numa_id 0 core_id 2:
packets received: 25902718
packet recirculations: 0
avg. datapath passes per packet: 1.00
emc hits: 25902697
megaflow hits: 0
avg. subtable lookups per megaflow hit: 0.00
miss with success upcall: 0
miss with failed upcall: 0
avg. packets per output batch: 28.11
idle cycles: 0 (0.00%)
processing cycles: 12138284463 (100.00%)
avg cycles per packet: 468.61 (12138284463/25902718)
avg processing cycles per packet: 468.61 (12138284463/25902718)

2018-05-18T22:34:57.865Z|1|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b40 was not found

2018-05-18T22:35:02.920Z|2|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a4584c was not found

2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident
set size after 10.4 seconds

2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
revalidators:1 rules:5 udpif keys:2

2018-05-18T22:35:21.910Z|3|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc1893e was not found

2018-05-18T22:35:21.924Z|4|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
flow creat error: 2 : message : flow rule creation failure


Looks like offloading didn't work and now the tput rate is lower,
which is not expected either.

I plan to continue on this and I appreciate if you have any idea of
what is going on.

Thanks,
fbl

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

--Shahaf


> -Original Message-
> From: Stokes, Ian <ian.sto...@intel.com>
> Sent: Monday, May 14, 2018 1:18 PM
> To: Olga Shern <ol...@mellanox.com>; Shahaf Shuler
> <shah...@mellanox.com>; f...@napatech.com
> Cc: ovs-dev@openvswitch.org; Flavio Leitner <f...@redhat.com>;
> simon.hor...@netronome.com; Kevin Traynor <ktray...@redhat.com>
> Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> > Hi Ian,
> >
> > It is taking almost a month to integrate this code.
> > Anything should be done to expedite it?
> 
> Hi Olga,
> 
> I realize it has been a month since the last version was submitted,
> unfortunately in terms of resources for reviews and testing the community is
> limited.
> 
> I flagged at the community call two weeks ago that there were a number of
> "large" feature patchsets currently on the backlog that could overlap. It was
> my intention to merge the detailed pmd metrics patchset first and then
> prioritize the HWOL series. There was no arguments raised against this
> approach.
> 
> The detailed PMD metrics was merged last week and HWOL is the feature I
> will look to upstream next.
> 
> In terms of expediting it, if the HWOL series could be rebased to head of
> master this week that would help, I can do final validation and have it as 
> part
> of the next pull request (this is aimed for the end of next week).

New v10[1] applied with rebase on top of latest master.

Hope to see this series on the next week pull request. 


[1]
https://patchwork.ozlabs.org/patch/916153/

> 
> Thanks
> Ian
> 
> >
> > Best regards,
> > Olga
> >
> > -Original Message-
> > From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev-
> > boun...@openvswitch.org] On Behalf Of Stokes, Ian
> > Sent: Thursday, May 10, 2018 10:19 PM
> > To: Shahaf Shuler <shah...@mellanox.com>; f...@napatech.com
> > Cc: ovs-dev@openvswitch.org; Flavio Leitner <f...@redhat.com>;
> > simon.hor...@netronome.com
> > Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with
> > rte_flow
> >
> >
> >
> > > -Original Message-
> > > From: Shahaf Shuler [mailto:shah...@mellanox.com]
> > > Sent: Thursday, May 10, 2018 1:41 PM
> > > To: Stokes, Ian <ian.sto...@intel.com>; f...@napatech.com
> > > Cc: ovs-dev@openvswitch.org; simon.hor...@netronome.com; Flavio
> > > Leitner <f...@redhat.com>
> > > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> > >
> > > Hi,
> > >
> > > Friday, April 20, 2018 11:07 AM, Stokes, Ian:
> > > > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> > > >
> > > > > > Hi,
> > > > > >
> > > > > > Here is a joint work from Mellanox and Napatech, to enable the
> > > > > > flow hw offload with the DPDK generic flow interface (rte_flow).
> > > > >
> > > > > Hi folks, I feel Mellanox/Netronome have reached the point where
> > > > > HWOL can be introduced to OVS DPDK pending performance review.
> > > >
> > > > Apologies , Mellanox an Napatech.
> > >
> > > Any progress with this series?
> > >
> >
> > Hi Shahaf,
> >
> > I'm finishing the pull request for this week now. This series is my
> > priority in the next pull request.
> >
> > The only outstanding item I need to do is a performance check before
> > and after to ensure there is no negative impact on the non HWOL
> > usecase. Other than that it should be ok.
> >
> > Thanks
> > Ian
> > >
> > > >
> > > > Ian
> > > > >
> > > > > This has not been an easy path, but I will put one final request
> > > > > out there for comments.
> > > > >
> > > > > If there are none then pending on Travis compilation and final
> > > > > performance checks I'm thinking of introducing this to the code
> > base.
> > > > >
> > > > > Keep in mind this feature is experimental, not only can it be
> > > > > changed but it can also be removed pending future review if not
> > > maintained.
> > > > >
> > > > > I invite all to give opinions here as I think this is an
> > > > > important feature.
> > > > >
> > > > > Sincere thanks to all involved.
> > > > >
> > > > > Regards
> > > > > Ian
> > > >

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Stokes, Ian]

> Hi Ian,
> 
> It is taking almost a month to integrate this code.
> Anything should be done to expedite it?

Hi Olga,

I realize it has been a month since the last version was submitted, 
unfortunately in terms of resources for reviews and testing the community is 
limited.

I flagged at the community call two weeks ago that there were a number of 
"large" feature patchsets currently on the backlog that could overlap. It was 
my intention to merge the detailed pmd metrics patchset first and then 
prioritize the HWOL series. There was no arguments raised against this approach.

The detailed PMD metrics was merged last week and HWOL is the feature I will 
look to upstream next.

In terms of expediting it, if the HWOL series could be rebased to head of 
master this week that would help, I can do final validation and have it as part 
of the next pull request (this is aimed for the end of next week).

Thanks
Ian

> 
> Best regards,
> Olga
> 
> -Original Message-
> From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev-
> boun...@openvswitch.org] On Behalf Of Stokes, Ian
> Sent: Thursday, May 10, 2018 10:19 PM
> To: Shahaf Shuler <shah...@mellanox.com>; f...@napatech.com
> Cc: ovs-dev@openvswitch.org; Flavio Leitner <f...@redhat.com>;
> simon.hor...@netronome.com
> Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> 
> 
> > -Original Message-
> > From: Shahaf Shuler [mailto:shah...@mellanox.com]
> > Sent: Thursday, May 10, 2018 1:41 PM
> > To: Stokes, Ian <ian.sto...@intel.com>; f...@napatech.com
> > Cc: ovs-dev@openvswitch.org; simon.hor...@netronome.com; Flavio
> > Leitner <f...@redhat.com>
> > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> >
> > Hi,
> >
> > Friday, April 20, 2018 11:07 AM, Stokes, Ian:
> > > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> > >
> > > > > Hi,
> > > > >
> > > > > Here is a joint work from Mellanox and Napatech, to enable the
> > > > > flow hw offload with the DPDK generic flow interface (rte_flow).
> > > >
> > > > Hi folks, I feel Mellanox/Netronome have reached the point where
> > > > HWOL can be introduced to OVS DPDK pending performance review.
> > >
> > > Apologies , Mellanox an Napatech.
> >
> > Any progress with this series?
> >
> 
> Hi Shahaf,
> 
> I'm finishing the pull request for this week now. This series is my
> priority in the next pull request.
> 
> The only outstanding item I need to do is a performance check before and
> after to ensure there is no negative impact on the non HWOL usecase. Other
> than that it should be ok.
> 
> Thanks
> Ian
> >
> > >
> > > Ian
> > > >
> > > > This has not been an easy path, but I will put one final request
> > > > out there for comments.
> > > >
> > > > If there are none then pending on Travis compilation and final
> > > > performance checks I'm thinking of introducing this to the code
> base.
> > > >
> > > > Keep in mind this feature is experimental, not only can it be
> > > > changed but it can also be removed pending future review if not
> > maintained.
> > > >
> > > > I invite all to give opinions here as I think this is an important
> > > > feature.
> > > >
> > > > Sincere thanks to all involved.
> > > >
> > > > Regards
> > > > Ian
> > > >
> > > >
> > > > >
> > > > > The basic idea is to associate the flow with a mark id (a
> > > > > unit32_t number).
> > > > > Later, we then get the flow directly from the mark id, which
> > > > > could bypass some heavy CPU operations, including but not
> > > > > limiting to mini flow extract, emc lookup, dpcls lookup, etc.
> > > > >
> > > > > The association is done with CMAP in patch 1. The CPU workload
> > > > > bypassing is done in patch 2. The flow offload is done in patch
> > > > > 3, which mainly does two things:
> > > > >
> > > > > - translate the ovs match to DPDK rte flow patterns
> > > > > - bind those patterns with a RSS + MARK action.
> > > > >
> > > > > Patch 5 makes the offload work happen in another thread, for
> > > > > leaving the datapath as light as possible.
> > > > >
> > > > > A PHY-PHY forwarding with 1000 mega fl

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Olga Shern]

Hi Ian, 

It is taking almost a month to integrate this code.
Anything should be done to expedite it?

Best regards,
Olga

-Original Message-
From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev-boun...@openvswitch.org] 
On Behalf Of Stokes, Ian
Sent: Thursday, May 10, 2018 10:19 PM
To: Shahaf Shuler <shah...@mellanox.com>; f...@napatech.com
Cc: ovs-dev@openvswitch.org; Flavio Leitner <f...@redhat.com>; 
simon.hor...@netronome.com
Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow



> -Original Message-
> From: Shahaf Shuler [mailto:shah...@mellanox.com]
> Sent: Thursday, May 10, 2018 1:41 PM
> To: Stokes, Ian <ian.sto...@intel.com>; f...@napatech.com
> Cc: ovs-dev@openvswitch.org; simon.hor...@netronome.com; Flavio 
> Leitner <f...@redhat.com>
> Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> Hi,
> 
> Friday, April 20, 2018 11:07 AM, Stokes, Ian:
> > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> >
> > > > Hi,
> > > >
> > > > Here is a joint work from Mellanox and Napatech, to enable the 
> > > > flow hw offload with the DPDK generic flow interface (rte_flow).
> > >
> > > Hi folks, I feel Mellanox/Netronome have reached the point where 
> > > HWOL can be introduced to OVS DPDK pending performance review.
> >
> > Apologies , Mellanox an Napatech.
> 
> Any progress with this series?
> 

Hi Shahaf,

I'm finishing the pull request for this week now. This series is my priority in 
the next pull request.

The only outstanding item I need to do is a performance check before and after 
to ensure there is no negative impact on the non HWOL usecase. Other than that 
it should be ok.

Thanks
Ian 
> 
> >
> > Ian
> > >
> > > This has not been an easy path, but I will put one final request 
> > > out there for comments.
> > >
> > > If there are none then pending on Travis compilation and final 
> > > performance checks I'm thinking of introducing this to the code base.
> > >
> > > Keep in mind this feature is experimental, not only can it be 
> > > changed but it can also be removed pending future review if not
> maintained.
> > >
> > > I invite all to give opinions here as I think this is an important 
> > > feature.
> > >
> > > Sincere thanks to all involved.
> > >
> > > Regards
> > > Ian
> > >
> > >
> > > >
> > > > The basic idea is to associate the flow with a mark id (a 
> > > > unit32_t number).
> > > > Later, we then get the flow directly from the mark id, which 
> > > > could bypass some heavy CPU operations, including but not 
> > > > limiting to mini flow extract, emc lookup, dpcls lookup, etc.
> > > >
> > > > The association is done with CMAP in patch 1. The CPU workload 
> > > > bypassing is done in patch 2. The flow offload is done in patch 
> > > > 3, which mainly does two things:
> > > >
> > > > - translate the ovs match to DPDK rte flow patterns
> > > > - bind those patterns with a RSS + MARK action.
> > > >
> > > > Patch 5 makes the offload work happen in another thread, for 
> > > > leaving the datapath as light as possible.
> > > >
> > > > A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) 
> > > > and
> > > > 1 million streams (tp_src=1000-1999, tp_dst=2000-2999) show more 
> > > > than 260% performance boost.
> > > >
> > > > Note that it's disabled by default, which can be enabled by:
> > > >
> > > > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
> > > >
> > > > v9: - introduced IP packet sanity checks in a seperate commit
> > > > - moved structs and enums definition to the begining of the file
> > > > - fixed sparse compilation error (error is assumed to be 
> > > > fixed,
> > > issues
> > > >   on Travis CI preventing from fully testing it.
> > > >
> > > > v8: - enhanced documentation with more details on supported
> protocols
> > > > - fixed VLOG to start with capital letter
> > > > - fixed compilation issues
> > > > - fixed coding style
> > > > - addressed the rest of Ian's comments
> > > >
> > > > v7: - fixed wrong hash for mark_to_flow that has been refactored 
> > > > in
> v6
> > > > - set the rss_conf

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Stokes, Ian]

> -Original Message-
> From: Shahaf Shuler [mailto:shah...@mellanox.com]
> Sent: Thursday, May 10, 2018 1:41 PM
> To: Stokes, Ian ; f...@napatech.com
> Cc: ovs-dev@openvswitch.org; simon.hor...@netronome.com; Flavio Leitner
> 
> Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> Hi,
> 
> Friday, April 20, 2018 11:07 AM, Stokes, Ian:
> > Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> >
> > > > Hi,
> > > >
> > > > Here is a joint work from Mellanox and Napatech, to enable the
> > > > flow hw offload with the DPDK generic flow interface (rte_flow).
> > >
> > > Hi folks, I feel Mellanox/Netronome have reached the point where
> > > HWOL can be introduced to OVS DPDK pending performance review.
> >
> > Apologies , Mellanox an Napatech.
> 
> Any progress with this series?
> 

Hi Shahaf,

I'm finishing the pull request for this week now. This series is my priority in 
the next pull request.

The only outstanding item I need to do is a performance check before and after 
to ensure there is no negative impact on the non HWOL usecase. Other than that 
it should be ok.

Thanks
Ian 
> 
> >
> > Ian
> > >
> > > This has not been an easy path, but I will put one final request out
> > > there for comments.
> > >
> > > If there are none then pending on Travis compilation and final
> > > performance checks I'm thinking of introducing this to the code base.
> > >
> > > Keep in mind this feature is experimental, not only can it be
> > > changed but it can also be removed pending future review if not
> maintained.
> > >
> > > I invite all to give opinions here as I think this is an important
> > > feature.
> > >
> > > Sincere thanks to all involved.
> > >
> > > Regards
> > > Ian
> > >
> > >
> > > >
> > > > The basic idea is to associate the flow with a mark id (a unit32_t
> > > > number).
> > > > Later, we then get the flow directly from the mark id, which could
> > > > bypass some heavy CPU operations, including but not limiting to
> > > > mini flow extract, emc lookup, dpcls lookup, etc.
> > > >
> > > > The association is done with CMAP in patch 1. The CPU workload
> > > > bypassing is done in patch 2. The flow offload is done in patch 3,
> > > > which mainly does two things:
> > > >
> > > > - translate the ovs match to DPDK rte flow patterns
> > > > - bind those patterns with a RSS + MARK action.
> > > >
> > > > Patch 5 makes the offload work happen in another thread, for
> > > > leaving the datapath as light as possible.
> > > >
> > > > A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999)
> > > > and
> > > > 1 million streams (tp_src=1000-1999, tp_dst=2000-2999) show more
> > > > than 260% performance boost.
> > > >
> > > > Note that it's disabled by default, which can be enabled by:
> > > >
> > > > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
> > > >
> > > > v9: - introduced IP packet sanity checks in a seperate commit
> > > > - moved structs and enums definition to the begining of the file
> > > > - fixed sparse compilation error (error is assumed to be
> > > > fixed,
> > > issues
> > > >   on Travis CI preventing from fully testing it.
> > > >
> > > > v8: - enhanced documentation with more details on supported
> protocols
> > > > - fixed VLOG to start with capital letter
> > > > - fixed compilation issues
> > > > - fixed coding style
> > > > - addressed the rest of Ian's comments
> > > >
> > > > v7: - fixed wrong hash for mark_to_flow that has been refactored in
> v6
> > > > - set the rss_conf for rss action to NULL, to workaround a
> > > > mlx5
> > > change
> > > >   in DPDK v17.11. Note that it will obey the rss settings
> > > > OVS-DPDK
> > > has
> > > >   set in the beginning. Thus, nothing should be effected.
> > > >
> > > > v6: - fixed a sparse warning
> > > > - added documentation
> > > > - used hash_int to compute mark to flow hash
> > > > - added more comments
> > > > - added lock for pot lookup
> > > > - rebased on top of the latest code
> > > >
> > > > v5: - fixed an issue that it took too long if we do flow add/remove
> > > >   repeatedly.
> > > > - removed an unused mutex lock
> > > > - turned most of the log level to DBG
> > > > - rebased on top of the latest code
> > > >
> > > > v4: - use RSS action instead of QUEUE action with MARK
> > > > - make it work with multiple queue (see patch 1)
> > > > - rebased on top of latest code
> > > >
> > > > v3: - The mark and id association is done with array instead of
> CMAP.
> > > > - Added a thread to do hw offload operations
> > > > - Removed macros completely
> > > > - dropped the patch to set FDIR_CONF, which is a workround some
> > > >   Intel NICs.
> > > > - Added a debug patch to show all flow patterns we have created.
> > > > - Misc fixes
> > > >
> > > > v2: - workaround the queue action issue
> > > > - fixed the tcp_flags being 

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Shahaf Shuler]

Hi,

Friday, April 20, 2018 11:07 AM, Stokes, Ian:
> Subject: RE: [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> > > Hi,
> > >
> > > Here is a joint work from Mellanox and Napatech, to enable the flow
> > > hw offload with the DPDK generic flow interface (rte_flow).
> >
> > Hi folks, I feel Mellanox/Netronome have reached the point where HWOL
> > can be introduced to OVS DPDK pending performance review.
> 
> Apologies , Mellanox an Napatech.

Any progress with this series? 


> 
> Ian
> >
> > This has not been an easy path, but I will put one final request out
> > there for comments.
> >
> > If there are none then pending on Travis compilation and final
> > performance checks I'm thinking of introducing this to the code base.
> >
> > Keep in mind this feature is experimental, not only can it be changed
> > but it can also be removed pending future review if not maintained.
> >
> > I invite all to give opinions here as I think this is an important
> > feature.
> >
> > Sincere thanks to all involved.
> >
> > Regards
> > Ian
> >
> >
> > >
> > > The basic idea is to associate the flow with a mark id (a unit32_t
> > > number).
> > > Later, we then get the flow directly from the mark id, which could
> > > bypass some heavy CPU operations, including but not limiting to mini
> > > flow extract, emc lookup, dpcls lookup, etc.
> > >
> > > The association is done with CMAP in patch 1. The CPU workload
> > > bypassing is done in patch 2. The flow offload is done in patch 3,
> > > which mainly does two things:
> > >
> > > - translate the ovs match to DPDK rte flow patterns
> > > - bind those patterns with a RSS + MARK action.
> > >
> > > Patch 5 makes the offload work happen in another thread, for leaving
> > > the datapath as light as possible.
> > >
> > > A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and
> > > 1 million streams (tp_src=1000-1999, tp_dst=2000-2999) show more
> > > than 260% performance boost.
> > >
> > > Note that it's disabled by default, which can be enabled by:
> > >
> > > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
> > >
> > > v9: - introduced IP packet sanity checks in a seperate commit
> > > - moved structs and enums definition to the begining of the file
> > > - fixed sparse compilation error (error is assumed to be fixed,
> > issues
> > >   on Travis CI preventing from fully testing it.
> > >
> > > v8: - enhanced documentation with more details on supported protocols
> > > - fixed VLOG to start with capital letter
> > > - fixed compilation issues
> > > - fixed coding style
> > > - addressed the rest of Ian's comments
> > >
> > > v7: - fixed wrong hash for mark_to_flow that has been refactored in v6
> > > - set the rss_conf for rss action to NULL, to workaround a mlx5
> > change
> > >   in DPDK v17.11. Note that it will obey the rss settings
> > > OVS-DPDK
> > has
> > >   set in the beginning. Thus, nothing should be effected.
> > >
> > > v6: - fixed a sparse warning
> > > - added documentation
> > > - used hash_int to compute mark to flow hash
> > > - added more comments
> > > - added lock for pot lookup
> > > - rebased on top of the latest code
> > >
> > > v5: - fixed an issue that it took too long if we do flow add/remove
> > >   repeatedly.
> > > - removed an unused mutex lock
> > > - turned most of the log level to DBG
> > > - rebased on top of the latest code
> > >
> > > v4: - use RSS action instead of QUEUE action with MARK
> > > - make it work with multiple queue (see patch 1)
> > > - rebased on top of latest code
> > >
> > > v3: - The mark and id association is done with array instead of CMAP.
> > > - Added a thread to do hw offload operations
> > > - Removed macros completely
> > > - dropped the patch to set FDIR_CONF, which is a workround some
> > >   Intel NICs.
> > > - Added a debug patch to show all flow patterns we have created.
> > > - Misc fixes
> > >
> > > v2: - workaround the queue action issue
> > > - fixed the tcp_flags being skipped issue, which also fixed the
> > >   build warnings
> > > - fixed l2 patterns for Intel nic
> > > - Converted some macros to functions
> > > - did not hardcode the max number of flow/action
> > > - rebased on top of the latest code
> > >
> > > Thanks.
> > >
> > > ---
> > >
> > > Finn Christensen (1):
> > >   netdev-dpdk: implement flow offload with rte flow
> > >
> > > Yuanhan Liu (6):
> > >   dpif-netdev: associate flow with a mark id
> > >   flow: Introduce IP packet sanity checks
> > >   dpif-netdev: retrieve flow directly from the flow mark
> > >   netdev-dpdk: add debug for rte flow patterns
> > >   dpif-netdev: do hw flow offload in a thread
> > >   Documentation: document ovs-dpdk flow offload
> > >
> > >  Documentation/howto/dpdk.rst |  22 ++
> > >  NEWS |   3 +-
> > >  lib/dp-packet.h  |  13 +
> > >  lib/dpif-netdev.c   

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Stokes, Ian]

> > Hi,
> >
> > Here is a joint work from Mellanox and Napatech, to enable the flow hw
> > offload with the DPDK generic flow interface (rte_flow).
> 
> Hi folks, I feel Mellanox/Netronome have reached the point where HWOL can
> be introduced to OVS DPDK pending performance review.

Apologies , Mellanox an Napatech.

Ian
> 
> This has not been an easy path, but I will put one final request out there
> for comments.
> 
> If there are none then pending on Travis compilation and final performance
> checks I'm thinking of introducing this to the code base.
> 
> Keep in mind this feature is experimental, not only can it be changed but
> it can also be removed pending future review if not maintained.
> 
> I invite all to give opinions here as I think this is an important
> feature.
> 
> Sincere thanks to all involved.
> 
> Regards
> Ian
> 
> 
> >
> > The basic idea is to associate the flow with a mark id (a unit32_t
> > number).
> > Later, we then get the flow directly from the mark id, which could
> > bypass some heavy CPU operations, including but not limiting to mini
> > flow extract, emc lookup, dpcls lookup, etc.
> >
> > The association is done with CMAP in patch 1. The CPU workload
> > bypassing is done in patch 2. The flow offload is done in patch 3,
> > which mainly does two things:
> >
> > - translate the ovs match to DPDK rte flow patterns
> > - bind those patterns with a RSS + MARK action.
> >
> > Patch 5 makes the offload work happen in another thread, for leaving
> > the datapath as light as possible.
> >
> > A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1
> > million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than
> > 260% performance boost.
> >
> > Note that it's disabled by default, which can be enabled by:
> >
> > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
> >
> > v9: - introduced IP packet sanity checks in a seperate commit
> > - moved structs and enums definition to the begining of the file
> > - fixed sparse compilation error (error is assumed to be fixed,
> issues
> >   on Travis CI preventing from fully testing it.
> >
> > v8: - enhanced documentation with more details on supported protocols
> > - fixed VLOG to start with capital letter
> > - fixed compilation issues
> > - fixed coding style
> > - addressed the rest of Ian's comments
> >
> > v7: - fixed wrong hash for mark_to_flow that has been refactored in v6
> > - set the rss_conf for rss action to NULL, to workaround a mlx5
> change
> >   in DPDK v17.11. Note that it will obey the rss settings OVS-DPDK
> has
> >   set in the beginning. Thus, nothing should be effected.
> >
> > v6: - fixed a sparse warning
> > - added documentation
> > - used hash_int to compute mark to flow hash
> > - added more comments
> > - added lock for pot lookup
> > - rebased on top of the latest code
> >
> > v5: - fixed an issue that it took too long if we do flow add/remove
> >   repeatedly.
> > - removed an unused mutex lock
> > - turned most of the log level to DBG
> > - rebased on top of the latest code
> >
> > v4: - use RSS action instead of QUEUE action with MARK
> > - make it work with multiple queue (see patch 1)
> > - rebased on top of latest code
> >
> > v3: - The mark and id association is done with array instead of CMAP.
> > - Added a thread to do hw offload operations
> > - Removed macros completely
> > - dropped the patch to set FDIR_CONF, which is a workround some
> >   Intel NICs.
> > - Added a debug patch to show all flow patterns we have created.
> > - Misc fixes
> >
> > v2: - workaround the queue action issue
> > - fixed the tcp_flags being skipped issue, which also fixed the
> >   build warnings
> > - fixed l2 patterns for Intel nic
> > - Converted some macros to functions
> > - did not hardcode the max number of flow/action
> > - rebased on top of the latest code
> >
> > Thanks.
> >
> > ---
> >
> > Finn Christensen (1):
> >   netdev-dpdk: implement flow offload with rte flow
> >
> > Yuanhan Liu (6):
> >   dpif-netdev: associate flow with a mark id
> >   flow: Introduce IP packet sanity checks
> >   dpif-netdev: retrieve flow directly from the flow mark
> >   netdev-dpdk: add debug for rte flow patterns
> >   dpif-netdev: do hw flow offload in a thread
> >   Documentation: document ovs-dpdk flow offload
> >
> >  Documentation/howto/dpdk.rst |  22 ++
> >  NEWS |   3 +-
> >  lib/dp-packet.h  |  13 +
> >  lib/dpif-netdev.c| 497 -
> >  lib/flow.c   | 155 ++--
> >  lib/flow.h   |   1 +
> >  lib/netdev-dpdk.c| 740
> +-
> >  lib/netdev.h |   6 +
> >  8 files changed, 1397 insertions(+), 40 deletions(-)
> >
> > --
> > 2.7.4
> 
> ___
> dev mailing list
> 

Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

[Stokes, Ian]

> Hi,
> 
> Here is a joint work from Mellanox and Napatech, to enable the flow hw
> offload with the DPDK generic flow interface (rte_flow).

Hi folks, I feel Mellanox/Netronome have reached the point where HWOL can be 
introduced to OVS DPDK pending performance review.

This has not been an easy path, but I will put one final request out there for 
comments.

If there are none then pending on Travis compilation and final performance 
checks I'm thinking of introducing this to the code base.

Keep in mind this feature is experimental, not only can it be changed but it 
can also be removed pending future review if not maintained.

I invite all to give opinions here as I think this is an important feature.

Sincere thanks to all involved.

Regards
Ian
 

> 
> The basic idea is to associate the flow with a mark id (a unit32_t
> number).
> Later, we then get the flow directly from the mark id, which could bypass
> some heavy CPU operations, including but not limiting to mini flow
> extract, emc lookup, dpcls lookup, etc.
> 
> The association is done with CMAP in patch 1. The CPU workload bypassing
> is done in patch 2. The flow offload is done in patch 3, which mainly does
> two things:
> 
> - translate the ovs match to DPDK rte flow patterns
> - bind those patterns with a RSS + MARK action.
> 
> Patch 5 makes the offload work happen in another thread, for leaving the
> datapath as light as possible.
> 
> A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1
> million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than 260%
> performance boost.
> 
> Note that it's disabled by default, which can be enabled by:
> 
> $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
> 
> v9: - introduced IP packet sanity checks in a seperate commit
> - moved structs and enums definition to the begining of the file
> - fixed sparse compilation error (error is assumed to be fixed, issues
>   on Travis CI preventing from fully testing it.
> 
> v8: - enhanced documentation with more details on supported protocols
> - fixed VLOG to start with capital letter
> - fixed compilation issues
> - fixed coding style
> - addressed the rest of Ian's comments
> 
> v7: - fixed wrong hash for mark_to_flow that has been refactored in v6
> - set the rss_conf for rss action to NULL, to workaround a mlx5 change
>   in DPDK v17.11. Note that it will obey the rss settings OVS-DPDK has
>   set in the beginning. Thus, nothing should be effected.
> 
> v6: - fixed a sparse warning
> - added documentation
> - used hash_int to compute mark to flow hash
> - added more comments
> - added lock for pot lookup
> - rebased on top of the latest code
> 
> v5: - fixed an issue that it took too long if we do flow add/remove
>   repeatedly.
> - removed an unused mutex lock
> - turned most of the log level to DBG
> - rebased on top of the latest code
> 
> v4: - use RSS action instead of QUEUE action with MARK
> - make it work with multiple queue (see patch 1)
> - rebased on top of latest code
> 
> v3: - The mark and id association is done with array instead of CMAP.
> - Added a thread to do hw offload operations
> - Removed macros completely
> - dropped the patch to set FDIR_CONF, which is a workround some
>   Intel NICs.
> - Added a debug patch to show all flow patterns we have created.
> - Misc fixes
> 
> v2: - workaround the queue action issue
> - fixed the tcp_flags being skipped issue, which also fixed the
>   build warnings
> - fixed l2 patterns for Intel nic
> - Converted some macros to functions
> - did not hardcode the max number of flow/action
> - rebased on top of the latest code
> 
> Thanks.
> 
> ---
> 
> Finn Christensen (1):
>   netdev-dpdk: implement flow offload with rte flow
> 
> Yuanhan Liu (6):
>   dpif-netdev: associate flow with a mark id
>   flow: Introduce IP packet sanity checks
>   dpif-netdev: retrieve flow directly from the flow mark
>   netdev-dpdk: add debug for rte flow patterns
>   dpif-netdev: do hw flow offload in a thread
>   Documentation: document ovs-dpdk flow offload
> 
>  Documentation/howto/dpdk.rst |  22 ++
>  NEWS |   3 +-
>  lib/dp-packet.h  |  13 +
>  lib/dpif-netdev.c| 497 -
>  lib/flow.c   | 155 ++--
>  lib/flow.h   |   1 +
>  lib/netdev-dpdk.c| 740 +-
>  lib/netdev.h |   6 +
>  8 files changed, 1397 insertions(+), 40 deletions(-)
> 
> --
> 2.7.4

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev