Re: [ovs-discuss] [OVN] logical flow explosion in lr_in_ip_input table for dnat_and_snat IPs

[Dumitru Ceara]

On 6/25/20 9:34 PM, Girish Moodalbail wrote:
> Hello Dumitru, Han,
> 
> So, we applied this patchset and gave it a spin on our large scale
> cluster and saw a significant reduction in the number of logical flows
> in lr_in_ip_input table. Before this patch there were around 1.6M flows
> in lr_in_ip_input table. However, after the patch we see about 26K
> flows. So that is significant reduction in number of logical flows.
> 
> In lr_in_ip_input, I see
> 
>   * priority 92 flows matching ARP requests for dnat_and_snat IPs on
> distributed gateway port with is_chassis_resident() and
> corresponding ARP reply
>   * priority 91 flows matching ARP requests for dnat_and_snat IPs on
> distributed gateway port with !is_chassis_resident() and
> corresponding drop
>   * priority 90 flow matching ARP request for dnat_and_snat IPs and
> corresponding ARP replies
> 
> So far so good. 

Hi Girish,

Great, thanks for testing out the series and confirming that it's
working ok.

> 
> However, not directly related to this patch per-se but directly related
> to the behaviour of ARP and dnat_and_snat IP, on the OVN chassis we are
> seeing a significant number of OpenFlow flows in table 27 (around 2.3M
> OpenFlow flows). This table gets populated from logical flows in
> table=19 (ls_in_l2_lkup) of logical switch.
> 
> The two logical flows in l2_in_l2_lkup that are contributing to huge
> number of OpenFlow flows are: (for the  entire logical flow entry,
> please
> see: https://gist.github.com/girishmg/57b3005030d421c59b30e6c36cfc9c18)
> 
> Priority=75 flow 
> =
> This flow looks like below (where 169.254.0.0/29 
> is dnat_and_snat subnet and 192.168.0.1 is the logical_switch's gateway IP)
> 
> table=19(ls_in_l2_lkup      ), priority=75   , match=(flags[1] == 0 &&
> arp.op == 1 && arp.tpa == { 169.254.3.107, 169.254.1.85, 192.168.0.1,
> 169.254.10.155, 169.254.1.6}), action=(outport = "stor-sdn-test1"; output;)
> 
> What this flow says is that any ARP request packet from the switch
> heading towards the default gateway or any of those 1-to-1 nat send it
> out through the port towards  the ovn_cluster_router’s ingress pipeline.
> Question though is why any Pod on the logical switch would send an ARP
> for an IP that is not in its subnet. A packet from a Pod towards a
> non-subnet IP should ARP only for the default gateway IP.
> 

This is a bug. I'll start working on a fix send a patch for it soon.

> Priority=80 Flow
> =
> This flow looks like below
> 
> table=19(ls_in_l2_lkup      ), priority=80   , match=(eth.src == {
> 0a:58:c0:a8:00:01, 6a:93:f4:55:aa:a7, ae:92:2d:33:24:ea,
> ba:0a:d3:7d:bc:e8, b2:2f:40:4d:d9:2b} && (arp.op == 1 || nd_ns)),
> action=(outport = "_MC_flood"; output;)
> 
> The question again for this flow is why will there be a self-originated
> arp requests for the dnat_and_snat IPs from inside of the node's logical
> switch. I can see how this is a possibility on the switch that has
> `localnet port` on it and to which the distributed router connects to
> through a gateway port. 
> 

This is also a bug, similar to the one above, we should only deal with
external_mac's that might be used on this port. I'll fix it too soon.

Thanks,
Dumitru

> Regards,
> ~Girish
> 
> On Wed, Jun 24, 2020 at 8:55 AM Dumitru Ceara  > wrote:
> 
> Hi Girish,
> 
> I sent a patch series to implement Han's suggestion:
> https://patchwork.ozlabs.org/project/openvswitch/list/?series=185580
> https://mail.openvswitch.org/pipermail/ovs-dev/2020-June/372005.html
> 
> It would be great if you could give it a run on your setup too.
> 
> Thanks,
> Dumitru
> 
> On 6/16/20 5:18 PM, Girish Moodalbail wrote:
> > Thanks Han for the update.
> >
> > Regards,
> > ~Girish 
> >
> > On Mon, Jun 15, 2020 at 12:55 PM Han Zhou  
> > >> wrote:
> >
> >     Sorry Girish, I can't promise for now. I will see if I have
> time in
> >     the next couple of weeks, but welcome anyone to volunteer on
> this if
> >     it is urgent.
> >
> >     On Mon, Jun 15, 2020 at 10:56 AM Girish Moodalbail
> >     mailto:gmoodalb...@gmail.com>
> >> wrote:
> >
> >         Hello Han,
> >
> >         On Wed, Jun 3, 2020 at 9:39 PM Han Zhou  
> >         >> wrote:
> >
> >
> >
> >             On Wed, Jun 3, 2020 at 7:16 PM Girish Moodalbail
> >             mailto:gmoodalb...@gmail.com>
> >> wrote:
> >
> >                 Hello all,
> >
> >                 While working on an extension, see the diagram
> below, to
> >                 the existing OVN 

Re: [ovs-discuss] [OVN] logical flow explosion in lr_in_ip_input table for dnat_and_snat IPs

[Girish Moodalbail]

Hello Dumitru, Han,

So, we applied this patchset and gave it a spin on our large scale cluster
and saw a significant reduction in the number of logical flows in
lr_in_ip_input table. Before this patch there were around 1.6M flows in
lr_in_ip_input table. However, after the patch we see about 26K flows. So
that is significant reduction in number of logical flows.

In lr_in_ip_input, I see

   - priority 92 flows matching ARP requests for dnat_and_snat IPs on
   distributed gateway port with is_chassis_resident() and corresponding ARP
   reply
   - priority 91 flows matching ARP requests for dnat_and_snat IPs on
   distributed gateway port with !is_chassis_resident() and corresponding drop
   - priority 90 flow matching ARP request for dnat_and_snat IPs and
   corresponding ARP replies

So far so good.

However, not directly related to this patch per-se but directly related to
the behaviour of ARP and dnat_and_snat IP, on the OVN chassis we are seeing
a significant number of OpenFlow flows in table 27 (around 2.3M OpenFlow
flows). This table gets populated from logical flows in table=19
(ls_in_l2_lkup) of logical switch.

The two logical flows in l2_in_l2_lkup that are contributing to huge number
of OpenFlow flows are: (for the  entire logical flow entry, please see:
https://gist.github.com/girishmg/57b3005030d421c59b30e6c36cfc9c18)

Priority=75 flow
=
This flow looks like below (where 169.254.0.0/29 is dnat_and_snat subnet
and 192.168.0.1 is the logical_switch's gateway IP)

table=19(ls_in_l2_lkup  ), priority=75   , match=(flags[1] == 0 &&
arp.op == 1 && arp.tpa == { 169.254.3.107, 169.254.1.85, 192.168.0.1,
169.254.10.155, 169.254.1.6}), action=(outport = "stor-sdn-test1"; output;)

What this flow says is that any ARP request packet from the switch heading
towards the default gateway or any of those 1-to-1 nat send it out through
the port towards  the ovn_cluster_router’s ingress pipeline. Question
though is why any Pod on the logical switch would send an ARP for an IP
that is not in its subnet. A packet from a Pod towards a non-subnet IP
should ARP only for the default gateway IP.

Priority=80 Flow
=
This flow looks like below

table=19(ls_in_l2_lkup  ), priority=80   , match=(eth.src == {
0a:58:c0:a8:00:01, 6a:93:f4:55:aa:a7, ae:92:2d:33:24:ea, ba:0a:d3:7d:bc:e8,
b2:2f:40:4d:d9:2b} && (arp.op == 1 || nd_ns)), action=(outport =
"_MC_flood"; output;)

The question again for this flow is why will there be a self-originated arp
requests for the dnat_and_snat IPs from inside of the node's logical
switch. I can see how this is a possibility on the switch that has
`localnet port` on it and to which the distributed router connects to
through a gateway port.

Regards,
~Girish

On Wed, Jun 24, 2020 at 8:55 AM Dumitru Ceara  wrote:

> Hi Girish,
>
> I sent a patch series to implement Han's suggestion:
> https://patchwork.ozlabs.org/project/openvswitch/list/?series=185580
> https://mail.openvswitch.org/pipermail/ovs-dev/2020-June/372005.html
>
> It would be great if you could give it a run on your setup too.
>
> Thanks,
> Dumitru
>
> On 6/16/20 5:18 PM, Girish Moodalbail wrote:
> > Thanks Han for the update.
> >
> > Regards,
> > ~Girish
> >
> > On Mon, Jun 15, 2020 at 12:55 PM Han Zhou  > > wrote:
> >
> > Sorry Girish, I can't promise for now. I will see if I have time in
> > the next couple of weeks, but welcome anyone to volunteer on this if
> > it is urgent.
> >
> > On Mon, Jun 15, 2020 at 10:56 AM Girish Moodalbail
> > mailto:gmoodalb...@gmail.com>> wrote:
> >
> > Hello Han,
> >
> > On Wed, Jun 3, 2020 at 9:39 PM Han Zhou  > > wrote:
> >
> >
> >
> > On Wed, Jun 3, 2020 at 7:16 PM Girish Moodalbail
> > mailto:gmoodalb...@gmail.com>>
> wrote:
> >
> > Hello all,
> >
> > While working on an extension, see the diagram below, to
> > the existing OVN logical topology for the ovn-kubernetes
> > project, I am seeing an explosion of the "Reply to ARP
> > requests" logical flows in the `lr_in_ip_input` table
> > for the distributed router (ovn_cluster_router)
> > configured with gateway port (rtol-LS)
> >
> > internet
> >-+-->
> > |
> > |
>
> >   +--localnet-port-+
> >   |LS  |
> >   +-ltor-LS+
> >|
> >|
> >  +-rtol-LS+
> >  |   ovn_cluster_router   |
> >  |  (Distributed 

Re: [ovs-discuss] HA OVN "Central" as a kubernetes service

[aginwala]

Hi:

There are a couple of options as I have been exploring this too:

1. Upstream ovn-k8s patches (
https://github.com/ovn-org/ovn-kubernetes/commit/a07b1a01af7e37b15c2e5f179ffad2b9f25a083d)
uses statefulset and headless service for starting ovn central raft cluster
with 3 replicas. Cluster startup code and pod specs are pretty neat that
addresses most of the doubts.

OVN components have been containerized too to start them in pods. You can
also refer to
https://github.com/ovn-org/ovn/blob/d6b56b1629d5984ef91864510f918e232efb89de/Documentation/intro/install/general.rst
 for the same and use them to make it work in pod specs too.


2. Write a new ovn operator similar to etcd operator
https://github.com/coreos/etcd-operator which just takes the count of raft
replicas and does the job in the background.

I also added ovn-k8s group so they can comment on any other ideas too. Hope
it helps.



On Thu, Jun 25, 2020 at 7:15 AM Brendan Doyle 
wrote:

> Hi,
>
> So I'm trying to find information on setting up an OVN "Central/Master"
> high availability (HA)
> Not as Active-Backup with Pacemaker, but as a cluster. But I want to
> deploy and manage that
> cluster as a Kubernetes service .
>
> I can find lots of stuff on "ovn-kube" but this seems to be using OVN as
> a  kubernetes CNI instead of
> Flannel etc.  But this is not what I want to do, I have a kubernetes
> cluster using Flannel as the CNI,
> now  I want to deploy a HA OVN "Central" as a kubernetes service. Kind
> of like how you can deploy
> a MySQL cluster in kubernetes using a SatefulSet deployment.
>
> I have found this:
>   https://github.com/ovn-org/ovn-kubernetes#readme
>
> But it is not clear to me if this is how to setup OVN as a kubernetes
> CNI or it's how to setup a HA
> OVN central as kubernetes service.
>
> Can anybody comment, has anyone done this?
>
>
> I guess I could run an OVN central as standalone and use a kubernetes
> deployment with 3
>   replica sets and "export" as a NodePort service. And have a
> floating/VIP on my kubernetes
> nodes. And direct ovn-controllers to the VIP. So only the pod that holds
> the VIP would service
> requests. This would work and give HA, but you don't get the performance
> of an OVN
> clustered Database Model, where each OVN central could service requests.
>
>
>
>
> Thanks
>
>
> Rdgs
> Brendan
>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] HA OVN "Central" as a kubernetes service

[Brendan Doyle]

Hi,

So I'm trying to find information on setting up an OVN "Central/Master" 
high availability (HA)
Not as Active-Backup with Pacemaker, but as a cluster. But I want to 
deploy and manage that

cluster as a Kubernetes service .

I can find lots of stuff on "ovn-kube" but this seems to be using OVN as 
a  kubernetes CNI instead of
Flannel etc.  But this is not what I want to do, I have a kubernetes 
cluster using Flannel as the CNI,
now  I want to deploy a HA OVN "Central" as a kubernetes service. Kind 
of like how you can deploy

a MySQL cluster in kubernetes using a SatefulSet deployment.

I have found this:
 https://github.com/ovn-org/ovn-kubernetes#readme

But it is not clear to me if this is how to setup OVN as a kubernetes 
CNI or it's how to setup a HA

OVN central as kubernetes service.

Can anybody comment, has anyone done this?


I guess I could run an OVN central as standalone and use a kubernetes 
deployment with 3
 replica sets and "export" as a NodePort service. And have a 
floating/VIP on my kubernetes
nodes. And direct ovn-controllers to the VIP. So only the pod that holds 
the VIP would service
requests. This would work and give HA, but you don't get the performance 
of an OVN

clustered Database Model, where each OVN central could service requests.




Thanks


Rdgs
Brendan

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss