Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Tue, Nov 13, 2018 at 09:19:19PM CET, michaels...@gmail.com wrote:
>Jiri,
>that seems to work.. Thanks.
>Let's upstream this patch
Done:
http://patchwork.ozlabs.org/patch/997403/
>
>Michael
>
>בתאריך יום ג׳, 13 בנוב׳ 2018 ב-12:13 מאת Jiri Pirko <j...@resnulli.us
>>:
>
>> Mon, Nov 12, 2018 at 09:22:03PM CET, michaels...@gmail.com wrote:
>> >Calling it from net_dev_init doesn't look clean.
>> >If you will see who is calling 'dev_add_offload' you will find out that
>> >there are couple of modules that doing that except of the 8021q. so either
>> >all modules should add offload from net_dev_init or stay with the current
>> >model where each module adds its offload.
>> >
>> >When adding a vlan interface using 'ip link add', the module 8021q is
>> >loaded. What is the flow that causing ip link to load this module?
>> >Why can't we do the same thing when a vlan interface is created from ovs?
>> >
>> >Jiri,
>> >Any other idea how we can solve this issue?
>>
>> Could you please test following patch? I will send it upstream once you
>> confirm it helps your case. Thanks!
>>
>> From: Jiri Pirko
>> Subject: [patch net-next RFC] net: 8021q: move vlan offload registrations
>> into
>> vlan_core
>>
>> Currently, the vlan packet offloads are registered only upon 8021q module
>> load. However, even without this module loaded, the offloads could be
>> utilized, for example by openvswitch datapath. As reported by Michael,
>> that causes 2x to 5x performance improvement, depending on a testcase.
>>
>> So move the vlan offload registrations into vlan_core and make this
>> available even without 8021q module loaded.
>>
>> Reported-by: Michael Shteinbok
>> Signed-off-by: Jiri Pirko
>> ---
>> net/8021q/vlan.c | 96 -
>> net/8021q/vlan_core.c | 99 +++
>> 2 files changed, 99 insertions(+), 96 deletions(-)
>>
>> diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
>> index 1b7a375c6616..aef1a977279c 100644
>> --- a/net/8021q/vlan.c
>> +++ b/net/8021q/vlan.c
>> @@ -648,93 +648,6 @@ static int vlan_ioctl_handler(struct net *net, void
>> __user *arg)
>> return err;
>> }
>>
>> -static struct sk_buff *vlan_gro_receive(struct list_head *head,
>> - struct sk_buff *skb)
>> -{
>> - const struct packet_offload *ptype;
>> - unsigned int hlen, off_vlan;
>> - struct sk_buff *pp = NULL;
>> - struct vlan_hdr *vhdr;
>> - struct sk_buff *p;
>> - __be16 type;
>> - int flush = 1;
>> -
>> - off_vlan = skb_gro_offset(skb);
>> - hlen = off_vlan + sizeof(*vhdr);
>> - vhdr = skb_gro_header_fast(skb, off_vlan);
>> - if (skb_gro_header_hard(skb, hlen)) {
>> - vhdr = skb_gro_header_slow(skb, hlen, off_vlan);
>> - if (unlikely(!vhdr))
>> - goto out;
>> - }
>> -
>> - type = vhdr->h_vlan_encapsulated_proto;
>> -
>> - rcu_read_lock();
>> - ptype = gro_find_receive_by_type(type);
>> - if (!ptype)
>> - goto out_unlock;
>> -
>> - flush = 0;
>> -
>> - list_for_each_entry(p, head, list) {
>> - struct vlan_hdr *vhdr2;
>> -
>> - if (!NAPI_GRO_CB(p)->same_flow)
>> - continue;
>> -
>> - vhdr2 = (struct vlan_hdr *)(p->data + off_vlan);
>> - if (compare_vlan_header(vhdr, vhdr2))
>> - NAPI_GRO_CB(p)->same_flow = 0;
>> - }
>> -
>> - skb_gro_pull(skb, sizeof(*vhdr));
>> - skb_gro_postpull_rcsum(skb, vhdr, sizeof(*vhdr));
>> - pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
>> -
>> -out_unlock:
>> - rcu_read_unlock();
>> -out:
>> - skb_gro_flush_final(skb, pp, flush);
>> -
>> - return pp;
>> -}
>> -
>> -static int vlan_gro_complete(struct sk_buff *skb, int nhoff)
>> -{
>> - struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);
>> - __be16 type = vhdr->h_vlan_encapsulated_proto;
>> - struct packet_offload *ptype;
>> - int err = -ENOENT;
>> -
>> - rcu_read_lock();
>> - ptype = gro_find_complete_by_type(type);
>> - if (ptype)
>> - err = ptype->callbacks.gro_complete(skb, nhoff +
>> sizeof(*vhdr));
>> -
>> - rcu_read_unlock();
>> - return err;
>> -}
>> -
>> -static struct packet_offload vlan_packet_offloads[] __read_mostly = {
>> - {
>> - .type = cpu_to_be16(ETH_P_8021Q),
>> - .priority = 10,
>> - .callbacks = {
>> - .gro_receive = vlan_gro_receive,
>> - .gro_complete = vlan_gro_complete,
>> - },
>> - },
>> - {
>> - .type = cpu_to_be16(ETH_P_8021AD),
>> - .priority = 10,
>> - .callbacks = {
>> - .gro_receive = vlan_gro_receive,
>> -
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Jiri,
that seems to work.. Thanks.
Let's upstream this patch
Michael
בתאריך יום ג׳, 13 בנוב׳ 2018 ב-12:13 מאת Jiri Pirko <j...@resnulli.us
>:
> Mon, Nov 12, 2018 at 09:22:03PM CET, michaels...@gmail.com wrote:
> >Calling it from net_dev_init doesn't look clean.
> >If you will see who is calling 'dev_add_offload' you will find out that
> >there are couple of modules that doing that except of the 8021q. so either
> >all modules should add offload from net_dev_init or stay with the current
> >model where each module adds its offload.
> >
> >When adding a vlan interface using 'ip link add', the module 8021q is
> >loaded. What is the flow that causing ip link to load this module?
> >Why can't we do the same thing when a vlan interface is created from ovs?
> >
> >Jiri,
> >Any other idea how we can solve this issue?
>
> Could you please test following patch? I will send it upstream once you
> confirm it helps your case. Thanks!
>
> From: Jiri Pirko
> Subject: [patch net-next RFC] net: 8021q: move vlan offload registrations
> into
> vlan_core
>
> Currently, the vlan packet offloads are registered only upon 8021q module
> load. However, even without this module loaded, the offloads could be
> utilized, for example by openvswitch datapath. As reported by Michael,
> that causes 2x to 5x performance improvement, depending on a testcase.
>
> So move the vlan offload registrations into vlan_core and make this
> available even without 8021q module loaded.
>
> Reported-by: Michael Shteinbok
> Signed-off-by: Jiri Pirko
> ---
> net/8021q/vlan.c | 96 -
> net/8021q/vlan_core.c | 99 +++
> 2 files changed, 99 insertions(+), 96 deletions(-)
>
> diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
> index 1b7a375c6616..aef1a977279c 100644
> --- a/net/8021q/vlan.c
> +++ b/net/8021q/vlan.c
> @@ -648,93 +648,6 @@ static int vlan_ioctl_handler(struct net *net, void
> __user *arg)
> return err;
> }
>
> -static struct sk_buff *vlan_gro_receive(struct list_head *head,
> - struct sk_buff *skb)
> -{
> - const struct packet_offload *ptype;
> - unsigned int hlen, off_vlan;
> - struct sk_buff *pp = NULL;
> - struct vlan_hdr *vhdr;
> - struct sk_buff *p;
> - __be16 type;
> - int flush = 1;
> -
> - off_vlan = skb_gro_offset(skb);
> - hlen = off_vlan + sizeof(*vhdr);
> - vhdr = skb_gro_header_fast(skb, off_vlan);
> - if (skb_gro_header_hard(skb, hlen)) {
> - vhdr = skb_gro_header_slow(skb, hlen, off_vlan);
> - if (unlikely(!vhdr))
> - goto out;
> - }
> -
> - type = vhdr->h_vlan_encapsulated_proto;
> -
> - rcu_read_lock();
> - ptype = gro_find_receive_by_type(type);
> - if (!ptype)
> - goto out_unlock;
> -
> - flush = 0;
> -
> - list_for_each_entry(p, head, list) {
> - struct vlan_hdr *vhdr2;
> -
> - if (!NAPI_GRO_CB(p)->same_flow)
> - continue;
> -
> - vhdr2 = (struct vlan_hdr *)(p->data + off_vlan);
> - if (compare_vlan_header(vhdr, vhdr2))
> - NAPI_GRO_CB(p)->same_flow = 0;
> - }
> -
> - skb_gro_pull(skb, sizeof(*vhdr));
> - skb_gro_postpull_rcsum(skb, vhdr, sizeof(*vhdr));
> - pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
> -
> -out_unlock:
> - rcu_read_unlock();
> -out:
> - skb_gro_flush_final(skb, pp, flush);
> -
> - return pp;
> -}
> -
> -static int vlan_gro_complete(struct sk_buff *skb, int nhoff)
> -{
> - struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);
> - __be16 type = vhdr->h_vlan_encapsulated_proto;
> - struct packet_offload *ptype;
> - int err = -ENOENT;
> -
> - rcu_read_lock();
> - ptype = gro_find_complete_by_type(type);
> - if (ptype)
> - err = ptype->callbacks.gro_complete(skb, nhoff +
> sizeof(*vhdr));
> -
> - rcu_read_unlock();
> - return err;
> -}
> -
> -static struct packet_offload vlan_packet_offloads[] __read_mostly = {
> - {
> - .type = cpu_to_be16(ETH_P_8021Q),
> - .priority = 10,
> - .callbacks = {
> - .gro_receive = vlan_gro_receive,
> - .gro_complete = vlan_gro_complete,
> - },
> - },
> - {
> - .type = cpu_to_be16(ETH_P_8021AD),
> - .priority = 10,
> - .callbacks = {
> - .gro_receive = vlan_gro_receive,
> - .gro_complete = vlan_gro_complete,
> - },
> - },
> -};
> -
> static int __net_init vlan_init_net(struct net *net)
> {
> struct vlan_net *vn = net_generic(net, vlan_net_id);
> @@ -762,7 +675,6 @@ static struct pernet_operations
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Mon, Nov 12, 2018 at 09:22:03PM CET, michaels...@gmail.com wrote:
>Calling it from net_dev_init doesn't look clean.
>If you will see who is calling 'dev_add_offload' you will find out that
>there are couple of modules that doing that except of the 8021q. so either
>all modules should add offload from net_dev_init or stay with the current
>model where each module adds its offload.
>
>When adding a vlan interface using 'ip link add', the module 8021q is
>loaded. What is the flow that causing ip link to load this module?
>Why can't we do the same thing when a vlan interface is created from ovs?
>
>Jiri,
>Any other idea how we can solve this issue?
Could you please test following patch? I will send it upstream once you
confirm it helps your case. Thanks!
From: Jiri Pirko
Subject: [patch net-next RFC] net: 8021q: move vlan offload registrations into
vlan_core
Currently, the vlan packet offloads are registered only upon 8021q module
load. However, even without this module loaded, the offloads could be
utilized, for example by openvswitch datapath. As reported by Michael,
that causes 2x to 5x performance improvement, depending on a testcase.
So move the vlan offload registrations into vlan_core and make this
available even without 8021q module loaded.
Reported-by: Michael Shteinbok
Signed-off-by: Jiri Pirko
---
net/8021q/vlan.c | 96 -
net/8021q/vlan_core.c | 99 +++
2 files changed, 99 insertions(+), 96 deletions(-)
diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
index 1b7a375c6616..aef1a977279c 100644
--- a/net/8021q/vlan.c
+++ b/net/8021q/vlan.c
@@ -648,93 +648,6 @@ static int vlan_ioctl_handler(struct net *net, void __user
*arg)
return err;
}
-static struct sk_buff *vlan_gro_receive(struct list_head *head,
- struct sk_buff *skb)
-{
- const struct packet_offload *ptype;
- unsigned int hlen, off_vlan;
- struct sk_buff *pp = NULL;
- struct vlan_hdr *vhdr;
- struct sk_buff *p;
- __be16 type;
- int flush = 1;
-
- off_vlan = skb_gro_offset(skb);
- hlen = off_vlan + sizeof(*vhdr);
- vhdr = skb_gro_header_fast(skb, off_vlan);
- if (skb_gro_header_hard(skb, hlen)) {
- vhdr = skb_gro_header_slow(skb, hlen, off_vlan);
- if (unlikely(!vhdr))
- goto out;
- }
-
- type = vhdr->h_vlan_encapsulated_proto;
-
- rcu_read_lock();
- ptype = gro_find_receive_by_type(type);
- if (!ptype)
- goto out_unlock;
-
- flush = 0;
-
- list_for_each_entry(p, head, list) {
- struct vlan_hdr *vhdr2;
-
- if (!NAPI_GRO_CB(p)->same_flow)
- continue;
-
- vhdr2 = (struct vlan_hdr *)(p->data + off_vlan);
- if (compare_vlan_header(vhdr, vhdr2))
- NAPI_GRO_CB(p)->same_flow = 0;
- }
-
- skb_gro_pull(skb, sizeof(*vhdr));
- skb_gro_postpull_rcsum(skb, vhdr, sizeof(*vhdr));
- pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb);
-
-out_unlock:
- rcu_read_unlock();
-out:
- skb_gro_flush_final(skb, pp, flush);
-
- return pp;
-}
-
-static int vlan_gro_complete(struct sk_buff *skb, int nhoff)
-{
- struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff);
- __be16 type = vhdr->h_vlan_encapsulated_proto;
- struct packet_offload *ptype;
- int err = -ENOENT;
-
- rcu_read_lock();
- ptype = gro_find_complete_by_type(type);
- if (ptype)
- err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(*vhdr));
-
- rcu_read_unlock();
- return err;
-}
-
-static struct packet_offload vlan_packet_offloads[] __read_mostly = {
- {
- .type = cpu_to_be16(ETH_P_8021Q),
- .priority = 10,
- .callbacks = {
- .gro_receive = vlan_gro_receive,
- .gro_complete = vlan_gro_complete,
- },
- },
- {
- .type = cpu_to_be16(ETH_P_8021AD),
- .priority = 10,
- .callbacks = {
- .gro_receive = vlan_gro_receive,
- .gro_complete = vlan_gro_complete,
- },
- },
-};
-
static int __net_init vlan_init_net(struct net *net)
{
struct vlan_net *vn = net_generic(net, vlan_net_id);
@@ -762,7 +675,6 @@ static struct pernet_operations vlan_net_ops = {
static int __init vlan_proto_init(void)
{
int err;
- unsigned int i;
pr_info("%s v%s\n", vlan_fullname, vlan_version);
@@ -786,9 +698,6 @@ static int __init vlan_proto_init(void)
if (err < 0)
goto err5;
- for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++)
- dev_add_offload(_packet_offloads[i]);
-
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Calling it from net_dev_init doesn't look clean. If you will see who is calling 'dev_add_offload' you will find out that there are couple of modules that doing that except of the 8021q. so either all modules should add offload from net_dev_init or stay with the current model where each module adds its offload. When adding a vlan interface using 'ip link add', the module 8021q is loaded. What is the flow that causing ip link to load this module? Why can't we do the same thing when a vlan interface is created from ovs? Jiri, Any other idea how we can solve this issue? Michael בתאריך יום א׳, 28 באוק׳ 2018 ב-18:54 מאת Jiri Pirko <j...@resnulli.us >: > Sun, Oct 28, 2018 at 03:03:42PM CET, michaels...@gmail.com wrote: > >Jiri, > >I am not sure it would be simple to move the add_offload to vlan_Core.c as > >the add_offload should happen once. > > Just call it from net_dev_init() > > >in vlan.c it's done as part of module init but in vlan_core.c we are not > >initializing any model and making some logic for doing that once looks > >awkward to me. > >Why can't we just load the module (8021q) once a vlan interface is added > >via OVS? > > I have no clue how exactly would you like to do it... > > > > > > > >בתאריך יום ה׳, 25 באוק׳ 2018 ב-15:15 מאת Jiri Pirko < > j...@resnulli.us > >>: > > > >> Thu, Oct 25, 2018 at 01:34:04PM CEST, michaels...@gmail.com wrote: > >> >I'v used ftrace to see which paths we take in the kernel. > >> >I remind you that the packet (in my scenario) looks like the following: > >> >ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload > >> > > >> >When 8021q is not loaded: > >> >dev_gro_receive <-napi_gro_receive > >> >inet_gro_receive <-dev_gro_receive > >> >udp4_gro_receive <-inet_gro_receive > >> >udp_gro_receive <-udp4_gro_receive > >> >udp4_lib_lookup_skb <-udp_gro_receive > >> >__udp4_lib_lookup <-udp4_lib_lookup_skb > >> >compute_score <-__udp4_lib_lookup > >> >vxlan_gro_receive <-udp_gro_receive > >> >__pskb_pull_tail <-vxlan_gro_receive > >> >skb_copy_bits <-__pskb_pull_tail > >> >eth_gro_receive <-vxlan_gro_receive > >> >__pskb_pull_tail <-eth_gro_receive > >> >skb_copy_bits <-__pskb_pull_tail > >> >gro_find_receive_by_type <-eth_gro_receive > >> >netif_receive_skb_internal <-napi_gro_receive > >> >ktime_get_real <-netif_receive_skb_internal > >> >getnstimeofday64 <-ktime_get_real > >> >__getnstimeofday64 <-getnstimeofday64 > >> >skb_defer_rx_timestamp <-netif_receive_skb_internal > >> >classify <-skb_defer_rx_timestamp > >> >__netif_receive_skb <-netif_receive_skb_internal > >> >__netif_receive_skb_core <-__netif_receive_skb > >> >ip_rcv <-__netif_receive_skb_core > >> >nf_hook_slow <-ip_rcv > >> >nf_iterate <-nf_hook_slow > >> >ipv4_conntrack_defrag <-nf_iterate > >> >ipv4_conntrack_in <-nf_iterate > >> > > >> >When 8021q was loaded manually: > >> > > >> > > >> >inet_gro_receive <-dev_gro_receive > >> >classify <-skb_clone_tx_timestamp > >> >udp4_gro_receive <-inet_gro_receive > >> >_raw_spin_lock <-sch_direct_xmit > >> >udp_gro_receive <-udp4_gro_receive > >> >udp4_lib_lookup_skb <-udp_gro_receive > >> >local_bh_enable <-__dev_queue_xmit > >> >__udp4_lib_lookup <-udp4_lib_lookup_skb > >> >__local_bh_enable_ip <-local_bh_enable > >> >compute_score <-__udp4_lib_lookup > >> >local_bh_enable <-ip_finish_output > >> >vxlan_gro_receive <-udp_gro_receive > >> >__local_bh_enable_ip <-local_bh_enable > >> >__pskb_pull_tail <-vxlan_gro_receive > >> >skb_copy_bits <-__pskb_pull_tail > >> >local_bh_enable <-__dev_queue_xmit > >> >eth_gro_receive <-vxlan_gro_receive > >> >__local_bh_enable_ip <-local_bh_enable > >> >__pskb_pull_tail <-eth_gro_receive > >> >skb_copy_bits <-__pskb_pull_tail > >> >gro_find_receive_by_type <-eth_gro_receive > >> >local_bh_enable <-__dev_queue_xmit > >> >__local_bh_enable_ip <-local_bh_enable > >> >vlan_gro_receive <-eth_gro_receive > >> >__pskb_pull_tail <-vlan_gro_receive > >> >local_bh_enable <-ip_finish_output > >> >skb_copy_bits <-__pskb_pull_tail > >> >__local_bh_enable_ip <-local_bh_enable > >> >gro_find_receive_by_type <-vlan_gro_receive > >> >inet_gro_receive <-vlan_gro_receive > >> >kfree_skb_partial <-tcp_rcv_established > >> >__pskb_pull_tail <-inet_gro_receive > >> >skb_release_all <-kfree_skb_partial > >> >skb_copy_bits <-__pskb_pull_tail > >> >skb_release_head_state <-skb_release_all > >> >skb_release_data <-skb_release_all > >> >tcp4_gro_receive <-inet_gro_receive > >> >tcp_gro_receive <-tcp4_gro_receive > >> >put_page <-skb_release_data > >> > > >> >You can see that eth_gro_receive calls vlan_gro_recieve when 8021q > module > >> >is loaded and then performs ding gro_receive for the next layers as > well > >> >(inet, tcp), whereas on default scenario where 8021q is not loaded the > >> vlan > >> >device ndo cannot be used. > >> > >> Ah, got it. > >> > >> What happens is that during the module load, vlan_proto_init() registers > >> vlan_packet_offloads: > >> > >> for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++) >
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Sun, Oct 28, 2018 at 03:03:42PM CET, michaels...@gmail.com wrote: >Jiri, >I am not sure it would be simple to move the add_offload to vlan_Core.c as >the add_offload should happen once. Just call it from net_dev_init() >in vlan.c it's done as part of module init but in vlan_core.c we are not >initializing any model and making some logic for doing that once looks >awkward to me. >Why can't we just load the module (8021q) once a vlan interface is added >via OVS? I have no clue how exactly would you like to do it... > > >בתאריך יום ה׳, 25 באוק׳ 2018 ב-15:15 מאת Jiri Pirko <j...@resnulli.us >>: > >> Thu, Oct 25, 2018 at 01:34:04PM CEST, michaels...@gmail.com wrote: >> >I'v used ftrace to see which paths we take in the kernel. >> >I remind you that the packet (in my scenario) looks like the following: >> >ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload >> > >> >When 8021q is not loaded: >> >dev_gro_receive <-napi_gro_receive >> >inet_gro_receive <-dev_gro_receive >> >udp4_gro_receive <-inet_gro_receive >> >udp_gro_receive <-udp4_gro_receive >> >udp4_lib_lookup_skb <-udp_gro_receive >> >__udp4_lib_lookup <-udp4_lib_lookup_skb >> >compute_score <-__udp4_lib_lookup >> >vxlan_gro_receive <-udp_gro_receive >> >__pskb_pull_tail <-vxlan_gro_receive >> >skb_copy_bits <-__pskb_pull_tail >> >eth_gro_receive <-vxlan_gro_receive >> >__pskb_pull_tail <-eth_gro_receive >> >skb_copy_bits <-__pskb_pull_tail >> >gro_find_receive_by_type <-eth_gro_receive >> >netif_receive_skb_internal <-napi_gro_receive >> >ktime_get_real <-netif_receive_skb_internal >> >getnstimeofday64 <-ktime_get_real >> >__getnstimeofday64 <-getnstimeofday64 >> >skb_defer_rx_timestamp <-netif_receive_skb_internal >> >classify <-skb_defer_rx_timestamp >> >__netif_receive_skb <-netif_receive_skb_internal >> >__netif_receive_skb_core <-__netif_receive_skb >> >ip_rcv <-__netif_receive_skb_core >> >nf_hook_slow <-ip_rcv >> >nf_iterate <-nf_hook_slow >> >ipv4_conntrack_defrag <-nf_iterate >> >ipv4_conntrack_in <-nf_iterate >> > >> >When 8021q was loaded manually: >> > >> > >> >inet_gro_receive <-dev_gro_receive >> >classify <-skb_clone_tx_timestamp >> >udp4_gro_receive <-inet_gro_receive >> >_raw_spin_lock <-sch_direct_xmit >> >udp_gro_receive <-udp4_gro_receive >> >udp4_lib_lookup_skb <-udp_gro_receive >> >local_bh_enable <-__dev_queue_xmit >> >__udp4_lib_lookup <-udp4_lib_lookup_skb >> >__local_bh_enable_ip <-local_bh_enable >> >compute_score <-__udp4_lib_lookup >> >local_bh_enable <-ip_finish_output >> >vxlan_gro_receive <-udp_gro_receive >> >__local_bh_enable_ip <-local_bh_enable >> >__pskb_pull_tail <-vxlan_gro_receive >> >skb_copy_bits <-__pskb_pull_tail >> >local_bh_enable <-__dev_queue_xmit >> >eth_gro_receive <-vxlan_gro_receive >> >__local_bh_enable_ip <-local_bh_enable >> >__pskb_pull_tail <-eth_gro_receive >> >skb_copy_bits <-__pskb_pull_tail >> >gro_find_receive_by_type <-eth_gro_receive >> >local_bh_enable <-__dev_queue_xmit >> >__local_bh_enable_ip <-local_bh_enable >> >vlan_gro_receive <-eth_gro_receive >> >__pskb_pull_tail <-vlan_gro_receive >> >local_bh_enable <-ip_finish_output >> >skb_copy_bits <-__pskb_pull_tail >> >__local_bh_enable_ip <-local_bh_enable >> >gro_find_receive_by_type <-vlan_gro_receive >> >inet_gro_receive <-vlan_gro_receive >> >kfree_skb_partial <-tcp_rcv_established >> >__pskb_pull_tail <-inet_gro_receive >> >skb_release_all <-kfree_skb_partial >> >skb_copy_bits <-__pskb_pull_tail >> >skb_release_head_state <-skb_release_all >> >skb_release_data <-skb_release_all >> >tcp4_gro_receive <-inet_gro_receive >> >tcp_gro_receive <-tcp4_gro_receive >> >put_page <-skb_release_data >> > >> >You can see that eth_gro_receive calls vlan_gro_recieve when 8021q module >> >is loaded and then performs ding gro_receive for the next layers as well >> >(inet, tcp), whereas on default scenario where 8021q is not loaded the >> vlan >> >device ndo cannot be used. >> >> Ah, got it. >> >> What happens is that during the module load, vlan_proto_init() registers >> vlan_packet_offloads: >> >> for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++) >> dev_add_offload(_packet_offloads[i]); >> >> I think that it would make sense to move this and the related code >> to net/8021q/vlan_core.c >> Then the offloads will be there regardless if the vlan module is loaded >> or not. >> >> Please feel free to spin-up a patch, will be more than happy to review >> it. Thanks! >> >> Jiri >> >> >> >> > >> >בתאריך יום ה׳, 25 באוק׳ 2018 ב-9:40 מאת Jiri Pirko <j...@resnulli.us >> >>: >> > >> >> Wed, Oct 24, 2018 at 10:12:54PM CEST, michaels...@gmail.com wrote: >> >> >Hi, >> >> >I noticed that there is a performance issue when running traffic on a >> vlan >> >> >interface that was created by OVS. >> >> >If we create a bridge with a vlan interface, the 8021q module is not >> >> loaded. >> >> >Then when packets with a 8021q tag arrives, the linux stack can't use >> the >> >> >vlan ndos (such as gro_recieve) since there
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Jiri, I am not sure it would be simple to move the add_offload to vlan_Core.c as the add_offload should happen once. in vlan.c it's done as part of module init but in vlan_core.c we are not initializing any model and making some logic for doing that once looks awkward to me. Why can't we just load the module (8021q) once a vlan interface is added via OVS? בתאריך יום ה׳, 25 באוק׳ 2018 ב-15:15 מאת Jiri Pirko <j...@resnulli.us >: > Thu, Oct 25, 2018 at 01:34:04PM CEST, michaels...@gmail.com wrote: > >I'v used ftrace to see which paths we take in the kernel. > >I remind you that the packet (in my scenario) looks like the following: > >ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload > > > >When 8021q is not loaded: > >dev_gro_receive <-napi_gro_receive > >inet_gro_receive <-dev_gro_receive > >udp4_gro_receive <-inet_gro_receive > >udp_gro_receive <-udp4_gro_receive > >udp4_lib_lookup_skb <-udp_gro_receive > >__udp4_lib_lookup <-udp4_lib_lookup_skb > >compute_score <-__udp4_lib_lookup > >vxlan_gro_receive <-udp_gro_receive > >__pskb_pull_tail <-vxlan_gro_receive > >skb_copy_bits <-__pskb_pull_tail > >eth_gro_receive <-vxlan_gro_receive > >__pskb_pull_tail <-eth_gro_receive > >skb_copy_bits <-__pskb_pull_tail > >gro_find_receive_by_type <-eth_gro_receive > >netif_receive_skb_internal <-napi_gro_receive > >ktime_get_real <-netif_receive_skb_internal > >getnstimeofday64 <-ktime_get_real > >__getnstimeofday64 <-getnstimeofday64 > >skb_defer_rx_timestamp <-netif_receive_skb_internal > >classify <-skb_defer_rx_timestamp > >__netif_receive_skb <-netif_receive_skb_internal > >__netif_receive_skb_core <-__netif_receive_skb > >ip_rcv <-__netif_receive_skb_core > >nf_hook_slow <-ip_rcv > >nf_iterate <-nf_hook_slow > >ipv4_conntrack_defrag <-nf_iterate > >ipv4_conntrack_in <-nf_iterate > > > >When 8021q was loaded manually: > > > > > >inet_gro_receive <-dev_gro_receive > >classify <-skb_clone_tx_timestamp > >udp4_gro_receive <-inet_gro_receive > >_raw_spin_lock <-sch_direct_xmit > >udp_gro_receive <-udp4_gro_receive > >udp4_lib_lookup_skb <-udp_gro_receive > >local_bh_enable <-__dev_queue_xmit > >__udp4_lib_lookup <-udp4_lib_lookup_skb > >__local_bh_enable_ip <-local_bh_enable > >compute_score <-__udp4_lib_lookup > >local_bh_enable <-ip_finish_output > >vxlan_gro_receive <-udp_gro_receive > >__local_bh_enable_ip <-local_bh_enable > >__pskb_pull_tail <-vxlan_gro_receive > >skb_copy_bits <-__pskb_pull_tail > >local_bh_enable <-__dev_queue_xmit > >eth_gro_receive <-vxlan_gro_receive > >__local_bh_enable_ip <-local_bh_enable > >__pskb_pull_tail <-eth_gro_receive > >skb_copy_bits <-__pskb_pull_tail > >gro_find_receive_by_type <-eth_gro_receive > >local_bh_enable <-__dev_queue_xmit > >__local_bh_enable_ip <-local_bh_enable > >vlan_gro_receive <-eth_gro_receive > >__pskb_pull_tail <-vlan_gro_receive > >local_bh_enable <-ip_finish_output > >skb_copy_bits <-__pskb_pull_tail > >__local_bh_enable_ip <-local_bh_enable > >gro_find_receive_by_type <-vlan_gro_receive > >inet_gro_receive <-vlan_gro_receive > >kfree_skb_partial <-tcp_rcv_established > >__pskb_pull_tail <-inet_gro_receive > >skb_release_all <-kfree_skb_partial > >skb_copy_bits <-__pskb_pull_tail > >skb_release_head_state <-skb_release_all > >skb_release_data <-skb_release_all > >tcp4_gro_receive <-inet_gro_receive > >tcp_gro_receive <-tcp4_gro_receive > >put_page <-skb_release_data > > > >You can see that eth_gro_receive calls vlan_gro_recieve when 8021q module > >is loaded and then performs ding gro_receive for the next layers as well > >(inet, tcp), whereas on default scenario where 8021q is not loaded the > vlan > >device ndo cannot be used. > > Ah, got it. > > What happens is that during the module load, vlan_proto_init() registers > vlan_packet_offloads: > > for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++) > dev_add_offload(_packet_offloads[i]); > > I think that it would make sense to move this and the related code > to net/8021q/vlan_core.c > Then the offloads will be there regardless if the vlan module is loaded > or not. > > Please feel free to spin-up a patch, will be more than happy to review > it. Thanks! > > Jiri > > > > > > >בתאריך יום ה׳, 25 באוק׳ 2018 ב-9:40 מאת Jiri Pirko <j...@resnulli.us > >>: > > > >> Wed, Oct 24, 2018 at 10:12:54PM CEST, michaels...@gmail.com wrote: > >> >Hi, > >> >I noticed that there is a performance issue when running traffic on a > vlan > >> >interface that was created by OVS. > >> >If we create a bridge with a vlan interface, the 8021q module is not > >> loaded. > >> >Then when packets with a 8021q tag arrives, the linux stack can't use > the > >> >vlan ndos (such as gro_recieve) since there is no such vlan device. > >> >If I perform the same test after loading the 8021q module, I get 2x-5x > >> >better performance. > >> > >> Could you please describe why exacly do you think you see this increase? > >> > >> > >> >I personally test that using tunnels (vxlan) but I think the
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Thu, Oct 25, 2018 at 01:34:04PM CEST, michaels...@gmail.com wrote: >I'v used ftrace to see which paths we take in the kernel. >I remind you that the packet (in my scenario) looks like the following: >ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload > >When 8021q is not loaded: >dev_gro_receive <-napi_gro_receive >inet_gro_receive <-dev_gro_receive >udp4_gro_receive <-inet_gro_receive >udp_gro_receive <-udp4_gro_receive >udp4_lib_lookup_skb <-udp_gro_receive >__udp4_lib_lookup <-udp4_lib_lookup_skb >compute_score <-__udp4_lib_lookup >vxlan_gro_receive <-udp_gro_receive >__pskb_pull_tail <-vxlan_gro_receive >skb_copy_bits <-__pskb_pull_tail >eth_gro_receive <-vxlan_gro_receive >__pskb_pull_tail <-eth_gro_receive >skb_copy_bits <-__pskb_pull_tail >gro_find_receive_by_type <-eth_gro_receive >netif_receive_skb_internal <-napi_gro_receive >ktime_get_real <-netif_receive_skb_internal >getnstimeofday64 <-ktime_get_real >__getnstimeofday64 <-getnstimeofday64 >skb_defer_rx_timestamp <-netif_receive_skb_internal >classify <-skb_defer_rx_timestamp >__netif_receive_skb <-netif_receive_skb_internal >__netif_receive_skb_core <-__netif_receive_skb >ip_rcv <-__netif_receive_skb_core >nf_hook_slow <-ip_rcv >nf_iterate <-nf_hook_slow >ipv4_conntrack_defrag <-nf_iterate >ipv4_conntrack_in <-nf_iterate > >When 8021q was loaded manually: > > >inet_gro_receive <-dev_gro_receive >classify <-skb_clone_tx_timestamp >udp4_gro_receive <-inet_gro_receive >_raw_spin_lock <-sch_direct_xmit >udp_gro_receive <-udp4_gro_receive >udp4_lib_lookup_skb <-udp_gro_receive >local_bh_enable <-__dev_queue_xmit >__udp4_lib_lookup <-udp4_lib_lookup_skb >__local_bh_enable_ip <-local_bh_enable >compute_score <-__udp4_lib_lookup >local_bh_enable <-ip_finish_output >vxlan_gro_receive <-udp_gro_receive >__local_bh_enable_ip <-local_bh_enable >__pskb_pull_tail <-vxlan_gro_receive >skb_copy_bits <-__pskb_pull_tail >local_bh_enable <-__dev_queue_xmit >eth_gro_receive <-vxlan_gro_receive >__local_bh_enable_ip <-local_bh_enable >__pskb_pull_tail <-eth_gro_receive >skb_copy_bits <-__pskb_pull_tail >gro_find_receive_by_type <-eth_gro_receive >local_bh_enable <-__dev_queue_xmit >__local_bh_enable_ip <-local_bh_enable >vlan_gro_receive <-eth_gro_receive >__pskb_pull_tail <-vlan_gro_receive >local_bh_enable <-ip_finish_output >skb_copy_bits <-__pskb_pull_tail >__local_bh_enable_ip <-local_bh_enable >gro_find_receive_by_type <-vlan_gro_receive >inet_gro_receive <-vlan_gro_receive >kfree_skb_partial <-tcp_rcv_established >__pskb_pull_tail <-inet_gro_receive >skb_release_all <-kfree_skb_partial >skb_copy_bits <-__pskb_pull_tail >skb_release_head_state <-skb_release_all >skb_release_data <-skb_release_all >tcp4_gro_receive <-inet_gro_receive >tcp_gro_receive <-tcp4_gro_receive >put_page <-skb_release_data > >You can see that eth_gro_receive calls vlan_gro_recieve when 8021q module >is loaded and then performs ding gro_receive for the next layers as well >(inet, tcp), whereas on default scenario where 8021q is not loaded the vlan >device ndo cannot be used. Ah, got it. What happens is that during the module load, vlan_proto_init() registers vlan_packet_offloads: for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++) dev_add_offload(_packet_offloads[i]); I think that it would make sense to move this and the related code to net/8021q/vlan_core.c Then the offloads will be there regardless if the vlan module is loaded or not. Please feel free to spin-up a patch, will be more than happy to review it. Thanks! Jiri > >בתאריך יום ה׳, 25 באוק׳ 2018 ב-9:40 מאת Jiri Pirko <j...@resnulli.us >>: > >> Wed, Oct 24, 2018 at 10:12:54PM CEST, michaels...@gmail.com wrote: >> >Hi, >> >I noticed that there is a performance issue when running traffic on a vlan >> >interface that was created by OVS. >> >If we create a bridge with a vlan interface, the 8021q module is not >> loaded. >> >Then when packets with a 8021q tag arrives, the linux stack can't use the >> >vlan ndos (such as gro_recieve) since there is no such vlan device. >> >If I perform the same test after loading the 8021q module, I get 2x-5x >> >better performance. >> >> Could you please describe why exacly do you think you see this increase? >> >> >> >I personally test that using tunnels (vxlan) but I think the issue exists >> >even without the tunnel. >> > >> >*Creating a bridge and vlan interface:* >> >ovs-vsctl --no-wait add-br br_0 >> >ip link set br_0 up >> >ovs-vsctl --no-wait add-port br_0 ens5f0 >> >ovs-vsctl --no-wait add-port br_0 vlan10 tag=10 -- set interface vlan10 >> >type=internal >> >ip link set vlan10 up >> >ip addr add dev vlan10 >> > >> >run traffic (netperf on the vlan10 interface) and see the result. >> >Then do 'modprobe 8021q' and perform the same test -> you will see much >> >better numbers. >> > >> >I am suggesting to load 80021q module once vlan interface is added through >> >ovs. >> >similarly as a vlan interface would be created using 'ip link'. >> > >>
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
I'v used ftrace to see which paths we take in the kernel. I remind you that the packet (in my scenario) looks like the following: ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload When 8021q is not loaded: dev_gro_receive <-napi_gro_receive inet_gro_receive <-dev_gro_receive udp4_gro_receive <-inet_gro_receive udp_gro_receive <-udp4_gro_receive udp4_lib_lookup_skb <-udp_gro_receive __udp4_lib_lookup <-udp4_lib_lookup_skb compute_score <-__udp4_lib_lookup vxlan_gro_receive <-udp_gro_receive __pskb_pull_tail <-vxlan_gro_receive skb_copy_bits <-__pskb_pull_tail eth_gro_receive <-vxlan_gro_receive __pskb_pull_tail <-eth_gro_receive skb_copy_bits <-__pskb_pull_tail gro_find_receive_by_type <-eth_gro_receive netif_receive_skb_internal <-napi_gro_receive ktime_get_real <-netif_receive_skb_internal getnstimeofday64 <-ktime_get_real __getnstimeofday64 <-getnstimeofday64 skb_defer_rx_timestamp <-netif_receive_skb_internal classify <-skb_defer_rx_timestamp __netif_receive_skb <-netif_receive_skb_internal __netif_receive_skb_core <-__netif_receive_skb ip_rcv <-__netif_receive_skb_core nf_hook_slow <-ip_rcv nf_iterate <-nf_hook_slow ipv4_conntrack_defrag <-nf_iterate ipv4_conntrack_in <-nf_iterate When 8021q was loaded manually: inet_gro_receive <-dev_gro_receive classify <-skb_clone_tx_timestamp udp4_gro_receive <-inet_gro_receive _raw_spin_lock <-sch_direct_xmit udp_gro_receive <-udp4_gro_receive udp4_lib_lookup_skb <-udp_gro_receive local_bh_enable <-__dev_queue_xmit __udp4_lib_lookup <-udp4_lib_lookup_skb __local_bh_enable_ip <-local_bh_enable compute_score <-__udp4_lib_lookup local_bh_enable <-ip_finish_output vxlan_gro_receive <-udp_gro_receive __local_bh_enable_ip <-local_bh_enable __pskb_pull_tail <-vxlan_gro_receive skb_copy_bits <-__pskb_pull_tail local_bh_enable <-__dev_queue_xmit eth_gro_receive <-vxlan_gro_receive __local_bh_enable_ip <-local_bh_enable __pskb_pull_tail <-eth_gro_receive skb_copy_bits <-__pskb_pull_tail gro_find_receive_by_type <-eth_gro_receive local_bh_enable <-__dev_queue_xmit __local_bh_enable_ip <-local_bh_enable vlan_gro_receive <-eth_gro_receive __pskb_pull_tail <-vlan_gro_receive local_bh_enable <-ip_finish_output skb_copy_bits <-__pskb_pull_tail __local_bh_enable_ip <-local_bh_enable gro_find_receive_by_type <-vlan_gro_receive inet_gro_receive <-vlan_gro_receive kfree_skb_partial <-tcp_rcv_established __pskb_pull_tail <-inet_gro_receive skb_release_all <-kfree_skb_partial skb_copy_bits <-__pskb_pull_tail skb_release_head_state <-skb_release_all skb_release_data <-skb_release_all tcp4_gro_receive <-inet_gro_receive tcp_gro_receive <-tcp4_gro_receive put_page <-skb_release_data You can see that eth_gro_receive calls vlan_gro_recieve when 8021q module is loaded and then performs ding gro_receive for the next layers as well (inet, tcp), whereas on default scenario where 8021q is not loaded the vlan device ndo cannot be used. בתאריך יום ה׳, 25 באוק׳ 2018 ב-9:40 מאת Jiri Pirko <j...@resnulli.us >: > Wed, Oct 24, 2018 at 10:12:54PM CEST, michaels...@gmail.com wrote: > >Hi, > >I noticed that there is a performance issue when running traffic on a vlan > >interface that was created by OVS. > >If we create a bridge with a vlan interface, the 8021q module is not > loaded. > >Then when packets with a 8021q tag arrives, the linux stack can't use the > >vlan ndos (such as gro_recieve) since there is no such vlan device. > >If I perform the same test after loading the 8021q module, I get 2x-5x > >better performance. > > Could you please describe why exacly do you think you see this increase? > > > >I personally test that using tunnels (vxlan) but I think the issue exists > >even without the tunnel. > > > >*Creating a bridge and vlan interface:* > >ovs-vsctl --no-wait add-br br_0 > >ip link set br_0 up > >ovs-vsctl --no-wait add-port br_0 ens5f0 > >ovs-vsctl --no-wait add-port br_0 vlan10 tag=10 -- set interface vlan10 > >type=internal > >ip link set vlan10 up > >ip addr add dev vlan10 > > > >run traffic (netperf on the vlan10 interface) and see the result. > >Then do 'modprobe 8021q' and perform the same test -> you will see much > >better numbers. > > > >I am suggesting to load 80021q module once vlan interface is added through > >ovs. > >similarly as a vlan interface would be created using 'ip link'. > > > >Michael > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)
Wed, Oct 24, 2018 at 10:12:54PM CEST, michaels...@gmail.com wrote: >Hi, >I noticed that there is a performance issue when running traffic on a vlan >interface that was created by OVS. >If we create a bridge with a vlan interface, the 8021q module is not loaded. >Then when packets with a 8021q tag arrives, the linux stack can't use the >vlan ndos (such as gro_recieve) since there is no such vlan device. >If I perform the same test after loading the 8021q module, I get 2x-5x >better performance. Could you please describe why exacly do you think you see this increase? >I personally test that using tunnels (vxlan) but I think the issue exists >even without the tunnel. > >*Creating a bridge and vlan interface:* >ovs-vsctl --no-wait add-br br_0 >ip link set br_0 up >ovs-vsctl --no-wait add-port br_0 ens5f0 >ovs-vsctl --no-wait add-port br_0 vlan10 tag=10 -- set interface vlan10 >type=internal >ip link set vlan10 up >ip addr add dev vlan10 > >run traffic (netperf on the vlan10 interface) and see the result. >Then do 'modprobe 8021q' and perform the same test -> you will see much >better numbers. > >I am suggesting to load 80021q module once vlan interface is added through >ovs. >similarly as a vlan interface would be created using 'ip link'. > >Michael ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
