Great news! Thank you Numan! Do you think the Canonical guys can backport this commit to the Ubuntu cloud 'Xena' repository to release a package update?
Thanks, Roberto Em seg., 27 de jun. de 2022 às 17:33, Numan Siddique <[email protected]> escreveu: > On Mon, Jun 27, 2022 at 1:35 PM Numan Siddique <[email protected]> wrote: > > > > On Mon, Jun 27, 2022 at 12:25 PM Numan Siddique <[email protected]> wrote: > > > > > > On Mon, Jun 27, 2022 at 3:56 AM Dumitru Ceara <[email protected]> > wrote: > > > > > > > > On 6/24/22 21:50, Numan Siddique wrote: > > > > > On Fri, Jun 24, 2022 at 11:53 AM ROBERTO BARTZEN ACOSTA via > discuss < > > > > > [email protected]> wrote: > > > > > > > > > >> Hi Dumitru, > > > > >> > > > > > > > > Hi Roberto, > > > > > > > > >> I also think this issue is related to ovn-monitor-all=true but > I'm not > > > > >> sure about the CPU usage consequences of disabling this. > > > > >> > > > > >> In OVSDB the changes are tracked and applied to each client in > the IDL > > > > >> layer. The OVSDB_IDL_MONITOR is set by default, then the IDL > replicates the > > > > >> changes in the database. Therefore, an OVSDB-IDL transaction > modifies the > > > > >> contents of a database and the client requests information about > the > > > > >> incremental changes. > > > > >> > > > > >> The ovn-monitor-all set the condition clause as true in > update_sb_monitors > > > > >> function and enable the monitor for many database information, > such as: > > > > >> Port_Bindings rows for local interfaces and local datapaths; > Monitor > > > > >> Logical_Flow, MAC_Binding, Multicast_Group, and DNS tables for > local > > > > >> datapaths; Monitor Controller_Event rows for local chassis; etc. > > > > >> > > > > >> Using these conditions clauses allows ovn-monitor-all to filter > to only > > > > >> replicate when specific conditions are met. However, the default > behavior > > > > >> is different, when the IDL replicates a particular table in the > database, > > > > >> it replicates every row in the table. > > > > >> > > > > >> I would like to better understand the computational advantages of > the > > > > >> ovn-controller conditional replication clauses, and the risks of > not > > > > >> enabling this parameter in a large-scale solution. > > > > >> > > > > >> Best regards, > > > > >> Roberto > > > > >> > > > > >> > > > > > On large scale deployments, our testing has shown that - > ovn-monitor-all=false > > > > > puts a significant amount of CPU load to the Southbound > ovsdb-server as it > > > > > has to conditionally send the data to each ovn-controller. > > > > > And hence we added the option - ovn-monitor-all=true. Drawback of > this is > > > > > that an ovn-controller with ovn-monitor-all=true will get all the > DB > > > > > updates. But this is still better compared to slow southbound > ovsdb-server. > > > > > > > > > > > > > Another potential drawback of ovn-monitor-all=true is additional > network > > > > traffic (all clients need to get all updates). But, as Numan > mentioned > > > > above, in all our scale testing (both for OpenStack and OpenShift) > the > > > > impact of this seems to be less significant compared to the impact on > > > > the Southbound when ovn-monitor-all=false. > > > > > > > > > > > > Numan, to avoid using ovn-monitor-all=false as workaround in this > case, > > > > do you think we can port > > > > > https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0 > > > > to branch-21.09 and see if Xena can pick it up? > > > > > > > > > > Given that this commit has fixed a bug, I think we can backport to > > > older branches. Looks like its not backported to branch-22.03 as > > > well. > > > > Actually it's already backported till branch-21.12. I'll backport to > > branch-21.09 once the CI here passes - > > > https://github.com/numansiddique/ovn/runs/7077845291?check_suite_focus=true > > > > Backport to branch-21.09 done. > > Thanks > Numan > > > Thanks > > Numan > > > > > > > > > > I'll see if it can be backported easily. > > > > > > Numan > > > > > > > Thanks, > > > > Dumitru > > > > > > > > > Thanks > > > > > Numan > > > > > > > > > > > > > > > > > > > >> Em sex., 24 de jun. de 2022 às 12:13, Tiago Pires < > [email protected]> > > > > >> escreveu: > > > > >> > > > > >>> Hi Dumitru, > > > > >>> > > > > >>> I did a test and configuring ovn-monitor-all as false to solve > this > > > > >>> behaviour. > > > > >>> It seems the option I have now is to use it as a workaround > until I have > > > > >>> conditions to upgrade to Yoga that has OVN 22.03. > > > > >>> > > > > >>> Thank you for your help. > > > > >>> > > > > >>> Regards, > > > > >>> > > > > >>> Tiago Pires > > > > >>> > > > > >>> > > > > >>> Em sex., 24 de jun. de 2022 às 04:27, Dumitru Ceara < > [email protected]> > > > > >>> escreveu: > > > > >>> > > > > >>>> On 6/23/22 22:23, Tiago Pires wrote: > > > > >>>>> Hi all, > > > > >>>>> > > > > >>>> > > > > >>>> Hi Tiago, > > > > >>>> > > > > >>>>> I did some troubleshooting and I'm seeing this error > (ovs-vswitchd) > > > > >>>> always > > > > >>>>> when a VM is created in a Chassi: > > > > >>>>> 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open > network > > > > >>>> device > > > > >>>>> tap8a43df0c-fd (No such device) > > > > >>>>> 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added > > > > >>>> interface > > > > >>>>> tap8a43df0c-fd on port 51 > > > > >>>>> 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added > > > > >>>> interface > > > > >>>>> tap3200bf1c-20 on port 52 > > > > >>>>> > 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430 > > > > >>>>> flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes) > > > > >>>>> > > > > >>>> > > > > >>>> It doesn't look to me like there's anything to worry about from > these > > > > >>>> logs. > > > > >>>> > > > > >>>>> On this commit > > > > >>>>> > > > > >>>> > http://patchwork.ozlabs.org/project/ovn/patch/[email protected]/ > > > > >>>>> it solved something similar to my issue. It seems the > ovs-vswitchd is > > > > >>>>> missing some flows and when I run the recompute it fixes it. > > > > >>>>> So, to avoid this issue I'm testing at this moment to run the > recompute > > > > >>>>> through libvirt hook when a VM gets "started" status. > > > > >>>>> > > > > >>>> > > > > >>>> While this might "fix" the issue it's not really ideal. > ovn-controller > > > > >>>> should properly install the flows all the time. Otherwise it's > a bug. > > > > >>>> > > > > >>>>> Regards, > > > > >>>>> > > > > >>>>> Tiago Pires > > > > >>>>> > > > > >>>>> > > > > >>>>> Em qua., 22 de jun. de 2022 às 19:43, Tiago Pires < > [email protected]> > > > > >>>>> escreveu: > > > > >>>>> > > > > >>>>>> Hi all, > > > > >>>>>> > > > > >>>>>> I'm trying to understand a stranger's behaviour regarding to > > > > >>>>>> ovn-controller. > > > > >>>>>> In my setup I have OVN 21.09/ OVS 2.16 and Xena and sometimes > when a > > > > >>>> new > > > > >>>>>> VM is created, this VM can reach other VMs in east-west > traffic (even > > > > >>>> in > > > > >>>>>> differents Chassis) but it can't reach an external network > (e.g. > > > > >>>> Internet) > > > > >>>>>> through Chassi Gateway. > > > > >>>>>> I ran the following trace: > > > > >>>>>> # ovs-appctl ofproto/trace br-int > > > > >>>>>> > > > > >>>> > in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64 > > > > >>>>>> > > > > >>>>>> And I got this output: > > > > >>>>>> Final flow: > > > > >>>>>> > > > > >>>> > recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 > > > > >>>>>> Megaflow: > > > > >>>>>> > > > > >>>> > recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= > > > > >>>>>> 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no > > > > >>>>>> Datapath actions: > > > > >>>>>> > > > > >>>> > ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535)) > > > > >>>>>> It seems the Datapath is querying the controller and I did not > > > > >>>> understand > > > > >>>>>> the reason. > > > > >>>>>> > > > > >>>>>> So, I did an ovn-controller recompute (ovn-appctl -t > ovn-controller > > > > >>>>>> recompute) on the Chassi where the VM is placed to check if > it could > > > > >>>> change > > > > >>>>>> the behaviour and I could trace the packet with success and > the VM > > > > >>>> started > > > > >>>>>> to communicate with the Internet normally: > > > > >>>>>> Final flow: > > > > >>>>>> > > > > >>>> > recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 > > > > >>>>>> Megaflow: > > > > >>>>>> > > > > >>>> > recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xffffff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= > > > > >>>>>> > 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no > > > > >>>>>> Datapath actions: > > > > >>>>>> > > > > >>>> > ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2 > > > > >>>>>> The Datapath action is using the tunnel with the Chassi > Gateway. > > > > >>>>>> > > > > >>>>>> It happens always with new VMs but sometimes. After running > the > > > > >>>> recompute > > > > >>>>>> on the Chassi, I created additional VMs and this issue did > not happen. > > > > >>>>>> > > > > >>>>>> In my Chassi I have enable these parameters also: > > > > >>>>>> ovn-monitor-all="true" > > > > >>>>>> ovn-openflow-probe-interval="0" > > > > >>>>>> ovn-remote-probe-interval="180000" > > > > >>>>>> > > > > >>>>>> Do you know this behaviour could be bug related? > > > > >>>> > > > > >>>> This is most definitely a bug. > > > > >>>> > > > > >>>> Very likely it's the bug that was fixed in this commit: > > > > >>>> > > > > >>>> > https://github.com/ovn-org/ovn/commit/0a4e073f4124b58f1b21778ec2293bbc4180e3e0 > > > > >>>> > > > > >>>> The change is available in the 21.12 stable branch and later. > So you > > > > >>>> need to upgrade the OVN version in your OpenStack deployment to > > > > >>>> something that includes it. > > > > >>>> > > > > >>>> Hope this helps. > > > > >>>> > > > > >>>> Regards, > > > > >>>> Dumitru > > > > >>>> > > > > >>>>>> > > > > >>>>>> Tiago Pires > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> _______________________________________________ > > > > >>>>> discuss mailing list > > > > >>>>> [email protected] > > > > >>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > >>>> > > > > >>>> _______________________________________________ > > > > >>> discuss mailing list > > > > >>> [email protected] > > > > >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> *‘Esta mensagem é direcionada apenas para os endereços constantes > no > > > > >> cabeçalho inicial. Se você não está listado nos endereços > constantes no > > > > >> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > dessa > > > > >> mensagem e cuja cópia, encaminhamento e/ou execução das ações > citadas estão > > > > >> imediatamente anuladas e proibidas’.* > > > > >> > > > > >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis > para > > > > >> assegurar que nenhum vírus esteja presente nesse e-mail, a > empresa não > > > > >> poderá aceitar a responsabilidade por quaisquer perdas ou danos > causados > > > > >> por esse e-mail ou por seus anexos’.* > > > > >> _______________________________________________ > > > > >> discuss mailing list > > > > >> [email protected] > > > > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > >> > > > > > > > > > > > > > _______________________________________________ > > > > discuss mailing list > > > > [email protected] > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
