Re: [ovs-discuss] ovn-controller stranger behaviour

[Numan Siddique]

On Fri, Jun 24, 2022 at 11:53 AM ROBERTO BARTZEN ACOSTA via discuss <
ovs-discuss@openvswitch.org> wrote:

> Hi Dumitru,
>
> I also think this issue is related to ovn-monitor-all=true but I'm not
> sure about the CPU usage consequences of disabling this.
>
> In OVSDB the changes are tracked and applied to each client in the IDL
> layer. The OVSDB_IDL_MONITOR is set by default, then the IDL replicates the
> changes in the database. Therefore, an OVSDB-IDL transaction modifies the
> contents of a database and the client requests information about the
> incremental changes.
>
> The ovn-monitor-all set the condition clause as true in update_sb_monitors
> function and enable the monitor for many database information, such as:
> Port_Bindings rows for local interfaces and local datapaths; Monitor
> Logical_Flow, MAC_Binding, Multicast_Group, and DNS tables for local
> datapaths; Monitor Controller_Event rows for local chassis; etc.
>
> Using these conditions clauses allows ovn-monitor-all to filter to only
> replicate when specific conditions are met. However, the default behavior
> is different, when the IDL replicates a particular table in the database,
> it replicates every row in the table.
>
> I would like to better understand the computational advantages of the
> ovn-controller conditional replication clauses, and the risks of not
> enabling this parameter in a large-scale solution.
>
> Best regards,
> Roberto
>
>
On large scale deployments,  our testing has shown that - ovn-monitor-all=false
puts a significant amount of CPU load to the Southbound ovsdb-server as it
has to conditionally send the data to each ovn-controller.
And hence we added the option - ovn-monitor-all=true.  Drawback of this is
that an ovn-controller with ovn-monitor-all=true will get all the DB
updates.  But this is still better compared to slow southbound ovsdb-server.

Thanks
Numan



> Em sex., 24 de jun. de 2022 às 12:13, Tiago Pires 
> escreveu:
>
>> Hi Dumitru,
>>
>> I did a test and configuring ovn-monitor-all as false to solve this
>> behaviour.
>> It seems the option I have now is to use it as a workaround until I have
>> conditions to upgrade to Yoga that has OVN 22.03.
>>
>> Thank you for your help.
>>
>> Regards,
>>
>> Tiago Pires
>>
>>
>> Em sex., 24 de jun. de 2022 às 04:27, Dumitru Ceara 
>> escreveu:
>>
>>> On 6/23/22 22:23, Tiago Pires wrote:
>>> > Hi all,
>>> >
>>>
>>> Hi Tiago,
>>>
>>> > I did some troubleshooting and I'm seeing this error (ovs-vswitchd)
>>> always
>>> > when a VM is created in a Chassi:
>>> > 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network
>>> device
>>> > tap8a43df0c-fd (No such device)
>>> > 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added
>>> interface
>>> > tap8a43df0c-fd on port 51
>>> > 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added
>>> interface
>>> > tap3200bf1c-20 on port 52
>>> > 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430
>>> > flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes)
>>> >
>>>
>>> It doesn't look to me like there's anything to worry about from these
>>> logs.
>>>
>>> > On this commit
>>> >
>>> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/
>>> > it solved something similar to my issue. It seems the ovs-vswitchd is
>>> > missing some flows and when I run the recompute it fixes it.
>>> > So, to avoid this issue I'm testing at this moment to run the recompute
>>> > through libvirt hook when a VM gets "started" status.
>>> >
>>>
>>> While this might "fix" the issue it's not really ideal.  ovn-controller
>>> should properly install the flows all the time.  Otherwise it's a bug.
>>>
>>> > Regards,
>>> >
>>> > Tiago Pires
>>> >
>>> >
>>> > Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires 
>>> > escreveu:
>>> >
>>> >> Hi all,
>>> >>
>>> >> I'm trying to understand a stranger's behaviour regarding to
>>> >> ovn-controller.
>>> >> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a
>>> new
>>> >> VM is created, this VM can reach other VMs in east-west traffic (even
>>> in
>>> >> differents Chassis) but it can't reach an external network (e.g.
>>> Internet)
>>> >> through Chassi Gateway.
>>> >> I ran the following trace:
>>> >> # ovs-appctl ofproto/trace br-int
>>> >>
>>> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64
>>> >>
>>> >> And I got this output:
>>> >> Final flow:
>>> >>
>>> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>>> >> Megaflow:
>>> >>
>>> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>>> >> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,n

[ovs-discuss] Building OVS debian package with DPDK

[Włodarczyk Adrian via discuss]

Hi,

My goal is to build OVS packages for Debian based system for ARM with DPDK 
support.

I use docker buildx for that emulatin arm architecture. Build machine is amd64.

Problem is that after successful build I have following packages generated:

libopenvswitch_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-dbg_2.16.92-2~1.gbp1fdff1_arm64.deb
libopenvswitch-dev_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-ipsec_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-pki_2.16.92-2~1.gbp1fdff1_all.deb
openvswitch-switch_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-test_2.16.92-2~1.gbp1fdff1_all.deb
openvswitch-testcontroller_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-common_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-vtep_2.16.92-2~1.gbp1fdff1_arm64.deb
openvswitch-datapath-dkms_2.16.92-2~1.gbp1fdff1_all.deb
python3-openvswitch_2.16.92-2~1.gbp1fdff1_all.deb

I expect to have generated also package openvswitch-switch-dpdk like this one 
https://packages.debian.org/bullseye/openvswitch-switch-dpdk. Correct me if I 
am wrong with that expectation.
This is my parameters passed to configure: --with-dpdk passed with static and 
shared and results is the same.
RUN ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc 
--with-dpdk=static/shared

For me the output from configure script is strange because it look as follow, 
one step says that dpdk is enabled, linking works, and last step mentioning 
dpdk, says that is not enabled:
... #13 232.9 checking for struct tcf_t.firstuse... yes #13 242.9 checking 
whether dpdk is enabled... yes #13 243.0 checking for DPDK... yes #13 243.3 
checking for faulty pkg-config version... no ... #13 280.0 checking whether 
linking with dpdk works... yes ... ... #16 531.3 checking whether dpdk is 
enabled... no ...

I am sending my Dockerfile in attachment. In this Dockerfile I am installing 
dpdk using apt but I have also tried to build dpdk from source based on that 
tutorial (https://docs.openvswitch.org/en/latest/intro/install/dpdk/) and 
result is the same.



BR Adrian Włodarczyk

[cid:logo-exatel_89270747-2b4b-4cc8-8de3-1e4525b182ef.png]Adrian 
Włodarczyk
  Starszy Programista
EXATEL SA
ul. Perkuna 47
04-164 Warszawa   Dział Rozwoju SDN
www.exatel.pl T 573789898
[cid:twitter_ed2e6b28-affc-4d54-8084-a9048872e037.png]
 [cid:linkedin_1fd08337-df5a-4ada-9173-3cbd182f385c.png] 
  
[cid:stopka_mail_facebook_00c5b97e-5f16-4510-a320-cc2ffd0158c9.png] 
 M
  E adrian.wlodarc...@exatel.pl

Spółka wpisana do rejestru przedsiębiorców w Sądzie Rejonowym dla m. st. 
Warszawy, XIV Wydział Gospodarczy, pod numerem KRS: 044577, kapitał 
zakładowy: 576 854 559 PLN, kapitał opłacony w całości, NIP: 527-010-45-68, 
BDO: 000250055, EXATEL SA, ul. Perkuna 47, 04-164 Warszawa.

Niniejsza wiadomość jest własnością EXATEL SA i może zawierać informacje poufne 
i/lub prawnie chronione. Jeśli nie są Państwo właściwym adresatem (lub 
otrzymali Państwo tę wiadomość na skutek pomyłki), prosimy o tym fakcie 
niezwłocznie poinformować nadawcę i usunąć otrzymaną wiadomość. Każde 
nieautoryzowane kopiowanie, ujawnianie lub rozpowszechnianie załączonej 
informacji jest zabronione.
Informacja o przetwarzaniu danych osobowych: 
link

This e-mail is a property of EXATEL SA and may contain confidential and/or 
privileged information. If you are not the intended recipient (or have received 
this e-mail by mistake) please notify the sender immediately and destroy this 
e-mail. Any unauthorized copying, disclosure or distribution of the material in 
this e-mail is strictly forbidden.
Information on the processing of personal data: 
link


Dockerfile-DebianBullseye-arm-dpdk
Description: Dockerfile-DebianBullseye-arm-dpdk
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-controller stranger behaviour

[ROBERTO BARTZEN ACOSTA via discuss]

Hi Dumitru,

I also think this issue is related to ovn-monitor-all=true but I'm not sure
about the CPU usage consequences of disabling this.

In OVSDB the changes are tracked and applied to each client in the IDL
layer. The OVSDB_IDL_MONITOR is set by default, then the IDL replicates the
changes in the database. Therefore, an OVSDB-IDL transaction modifies the
contents of a database and the client requests information about the
incremental changes.

The ovn-monitor-all set the condition clause as true in update_sb_monitors
function and enable the monitor for many database information, such as:
Port_Bindings rows for local interfaces and local datapaths; Monitor
Logical_Flow, MAC_Binding, Multicast_Group, and DNS tables for local
datapaths; Monitor Controller_Event rows for local chassis; etc.

Using these conditions clauses allows ovn-monitor-all to filter to only
replicate when specific conditions are met. However, the default behavior
is different, when the IDL replicates a particular table in the database,
it replicates every row in the table.

I would like to better understand the computational advantages of the
ovn-controller conditional replication clauses, and the risks of not
enabling this parameter in a large-scale solution.

Best regards,
Roberto


Em sex., 24 de jun. de 2022 às 12:13, Tiago Pires 
escreveu:

> Hi Dumitru,
>
> I did a test and configuring ovn-monitor-all as false to solve this
> behaviour.
> It seems the option I have now is to use it as a workaround until I have
> conditions to upgrade to Yoga that has OVN 22.03.
>
> Thank you for your help.
>
> Regards,
>
> Tiago Pires
>
>
> Em sex., 24 de jun. de 2022 às 04:27, Dumitru Ceara 
> escreveu:
>
>> On 6/23/22 22:23, Tiago Pires wrote:
>> > Hi all,
>> >
>>
>> Hi Tiago,
>>
>> > I did some troubleshooting and I'm seeing this error (ovs-vswitchd)
>> always
>> > when a VM is created in a Chassi:
>> > 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network device
>> > tap8a43df0c-fd (No such device)
>> > 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added
>> interface
>> > tap8a43df0c-fd on port 51
>> > 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added
>> interface
>> > tap3200bf1c-20 on port 52
>> > 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430
>> > flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes)
>> >
>>
>> It doesn't look to me like there's anything to worry about from these
>> logs.
>>
>> > On this commit
>> >
>> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/
>> > it solved something similar to my issue. It seems the ovs-vswitchd is
>> > missing some flows and when I run the recompute it fixes it.
>> > So, to avoid this issue I'm testing at this moment to run the recompute
>> > through libvirt hook when a VM gets "started" status.
>> >
>>
>> While this might "fix" the issue it's not really ideal.  ovn-controller
>> should properly install the flows all the time.  Otherwise it's a bug.
>>
>> > Regards,
>> >
>> > Tiago Pires
>> >
>> >
>> > Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires 
>> > escreveu:
>> >
>> >> Hi all,
>> >>
>> >> I'm trying to understand a stranger's behaviour regarding to
>> >> ovn-controller.
>> >> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a
>> new
>> >> VM is created, this VM can reach other VMs in east-west traffic (even
>> in
>> >> differents Chassis) but it can't reach an external network (e.g.
>> Internet)
>> >> through Chassi Gateway.
>> >> I ran the following trace:
>> >> # ovs-appctl ofproto/trace br-int
>> >>
>> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64
>> >>
>> >> And I got this output:
>> >> Final flow:
>> >>
>> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>> >> Megaflow:
>> >>
>> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>> >> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no
>> >> Datapath actions:
>> >>
>> ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535))
>> >> It seems the Datapath is querying the controller and I did not
>> understand
>> >> the reason.
>> >>
>> >> So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller
>> >> recompute) on the Chassi where the VM is placed to check if it could
>> change
>> >> the behaviour and I could trace the packet with success and the VM
>> started
>> >> to communicate with the Internet normally:
>> >>

Re: [ovs-discuss] ovn-controller stranger behaviour

[Tiago Pires]

Hi Dumitru,

I did a test and configuring ovn-monitor-all as false to solve this
behaviour.
It seems the option I have now is to use it as a workaround until I have
conditions to upgrade to Yoga that has OVN 22.03.

Thank you for your help.

Regards,

Tiago Pires


Em sex., 24 de jun. de 2022 às 04:27, Dumitru Ceara 
escreveu:

> On 6/23/22 22:23, Tiago Pires wrote:
> > Hi all,
> >
>
> Hi Tiago,
>
> > I did some troubleshooting and I'm seeing this error (ovs-vswitchd)
> always
> > when a VM is created in a Chassi:
> > 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network device
> > tap8a43df0c-fd (No such device)
> > 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added interface
> > tap8a43df0c-fd on port 51
> > 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added interface
> > tap3200bf1c-20 on port 52
> > 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430
> > flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes)
> >
>
> It doesn't look to me like there's anything to worry about from these logs.
>
> > On this commit
> >
> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/
> > it solved something similar to my issue. It seems the ovs-vswitchd is
> > missing some flows and when I run the recompute it fixes it.
> > So, to avoid this issue I'm testing at this moment to run the recompute
> > through libvirt hook when a VM gets "started" status.
> >
>
> While this might "fix" the issue it's not really ideal.  ovn-controller
> should properly install the flows all the time.  Otherwise it's a bug.
>
> > Regards,
> >
> > Tiago Pires
> >
> >
> > Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires 
> > escreveu:
> >
> >> Hi all,
> >>
> >> I'm trying to understand a stranger's behaviour regarding to
> >> ovn-controller.
> >> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a new
> >> VM is created, this VM can reach other VMs in east-west traffic (even in
> >> differents Chassis) but it can't reach an external network (e.g.
> Internet)
> >> through Chassi Gateway.
> >> I ran the following trace:
> >> # ovs-appctl ofproto/trace br-int
> >>
> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64
> >>
> >> And I got this output:
> >> Final flow:
> >>
> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
> >> Megaflow:
> >>
> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
> >> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no
> >> Datapath actions:
> >>
> ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535))
> >> It seems the Datapath is querying the controller and I did not
> understand
> >> the reason.
> >>
> >> So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller
> >> recompute) on the Chassi where the VM is placed to check if it could
> change
> >> the behaviour and I could trace the packet with success and the VM
> started
> >> to communicate with the Internet normally:
> >> Final flow:
> >>
> recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
> >> Megaflow:
> >>
> recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
> >> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no
> >> Datapath actions:
> >>
> ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2
> >> The Datapath action is using the tunnel with the Chassi Gateway.
> >>
> >> It happens always with new VMs but sometimes. After running the
> recompute
> >> on the Chassi, I created additional VMs and this issue did not happen.
> >>
> >> In my Chassi I have enable these parameters also:
> >> ovn-monitor-all="true"
> >> ovn-openflow-probe-interval="0"
> >> ovn-remote-probe-interval="18"
> >>
> >> Do you know this behaviour could be bug related?
>
> This is most definitely a bug.
>
> Very likely it's the bug that was fixed in this commit:
>
> https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0
>
>

Re: [ovs-discuss] ovn-controller stranger behaviour

[Han Zhou]

Hi Tiago,

Thanks for reporting the problem. It seems you can easily reproduce the
problem, right? If so, could you enable debug log for ovn-controller before
triggering the recompute, and then we can see what flows are added during
recompute from the logs of the ofctrl module?

Thanks,
Han

On Thu, Jun 23, 2022 at 1:24 PM Tiago Pires  wrote:

> Hi all,
>
> I did some troubleshooting and I'm seeing this error (ovs-vswitchd) always
> when a VM is created in a Chassi:
> 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network device
> tap8a43df0c-fd (No such device)
> 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added interface
> tap8a43df0c-fd on port 51
> 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added interface
> tap3200bf1c-20 on port 52
> 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430
> flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes)
>
> On this commit
> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/
> it solved something similar to my issue. It seems the ovs-vswitchd is
> missing some flows and when I run the recompute it fixes it.
> So, to avoid this issue I'm testing at this moment to run the recompute
> through libvirt hook when a VM gets "started" status.
>
> Regards,
>
> Tiago Pires
>
>
> Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires 
> escreveu:
>
>> Hi all,
>>
>> I'm trying to understand a stranger's behaviour regarding to
>> ovn-controller.
>> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a new
>> VM is created, this VM can reach other VMs in east-west traffic (even in
>> differents Chassis) but it can't reach an external network (e.g. Internet)
>> through Chassi Gateway.
>> I ran the following trace:
>> # ovs-appctl ofproto/trace br-int
>> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64
>>
>> And I got this output:
>> Final flow:
>> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>> Megaflow:
>> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no
>> Datapath actions:
>> ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535))
>> It seems the Datapath is querying the controller and I did not understand
>> the reason.
>>
>> So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller
>> recompute) on the Chassi where the VM is placed to check if it could change
>> the behaviour and I could trace the packet with success and the VM started
>> to communicate with the Internet normally:
>> Final flow:
>> recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>> Megaflow:
>> recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no
>> Datapath actions:
>> ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2
>> The Datapath action is using the tunnel with the Chassi Gateway.
>>
>> It happens always with new VMs but sometimes. After running the recompute
>> on the Chassi, I created additional VMs and this issue did not happen.
>>
>> In my Chassi I have enable these parameters also:
>> ovn-monitor-all="true"
>> ovn-openflow-probe-interval="0"
>> ovn-remote-probe-interval="18"
>>
>> Do you know this behaviour could be bug related?
>>
>> Tiago Pires
>>
>>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovn-controller stranger behaviour

[Dumitru Ceara]

On 6/23/22 22:23, Tiago Pires wrote:
> Hi all,
> 

Hi Tiago,

> I did some troubleshooting and I'm seeing this error (ovs-vswitchd) always
> when a VM is created in a Chassi:
> 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network device
> tap8a43df0c-fd (No such device)
> 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added interface
> tap8a43df0c-fd on port 51
> 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added interface
> tap3200bf1c-20 on port 52
> 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430
> flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes)
> 

It doesn't look to me like there's anything to worry about from these logs.

> On this commit
> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dce...@redhat.com/
> it solved something similar to my issue. It seems the ovs-vswitchd is
> missing some flows and when I run the recompute it fixes it.
> So, to avoid this issue I'm testing at this moment to run the recompute
> through libvirt hook when a VM gets "started" status.
> 

While this might "fix" the issue it's not really ideal.  ovn-controller
should properly install the flows all the time.  Otherwise it's a bug.

> Regards,
> 
> Tiago Pires
> 
> 
> Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires 
> escreveu:
> 
>> Hi all,
>>
>> I'm trying to understand a stranger's behaviour regarding to
>> ovn-controller.
>> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes when a new
>> VM is created, this VM can reach other VMs in east-west traffic (even in
>> differents Chassis) but it can't reach an external network (e.g. Internet)
>> through Chassi Gateway.
>> I ran the following trace:
>> # ovs-appctl ofproto/trace br-int
>> in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64
>>
>> And I got this output:
>> Final flow:
>> recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>> Megaflow:
>> recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no
>> Datapath actions:
>> ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535))
>> It seems the Datapath is querying the controller and I did not understand
>> the reason.
>>
>> So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller
>> recompute) on the Chassi where the VM is placed to check if it could change
>> the behaviour and I could trace the packet with success and the VM started
>> to communicate with the Internet normally:
>> Final flow:
>> recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0
>> Megaflow:
>> recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=
>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no
>> Datapath actions:
>> ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2
>> The Datapath action is using the tunnel with the Chassi Gateway.
>>
>> It happens always with new VMs but sometimes. After running the recompute
>> on the Chassi, I created additional VMs and this issue did not happen.
>>
>> In my Chassi I have enable these parameters also:
>> ovn-monitor-all="true"
>> ovn-openflow-probe-interval="0"
>> ovn-remote-probe-interval="18"
>>
>> Do you know this behaviour could be bug related?

This is most definitely a bug.

Very likely it's the bug that was fixed in this commit:
https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0

The change is available in the 21.12 stable branch and later.  So you
need to upgrade the OVN version in your OpenStack deployment to
something that includes it.

Hope this helps.

Regards,
Dumitru

>>
>> Tiago Pires
>>
>>
> 
> 
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listin