On Thu, Aug 6, 2020 at 10:13 AM Han Zhou <zhou...@gmail.com> wrote: > > > On Thu, Aug 6, 2020 at 8:54 AM Venugopal Iyer <venugop...@nvidia.com> > wrote: > >> Hi, Han: >> >> >> >> A comment inline: >> >> >> >> *From:* ovn-kuberne...@googlegroups.com <ovn-kuberne...@googlegroups.com> >> *On Behalf Of *Han Zhou >> *Sent:* Wednesday, August 5, 2020 3:36 PM >> *To:* Winson Wang <windson.w...@gmail.com> >> *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com; >> Dumitru Ceara <dce...@redhat.com>; Han Zhou <hz...@ovn.org> >> *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process >> keep the previous Open Flow in br-int >> >> >> >> *External email: Use caution opening links or attachments* >> >> >> >> >> >> >> >> On Wed, Aug 5, 2020 at 12:58 PM Winson Wang <windson.w...@gmail.com> >> wrote: >> >> Hello OVN Experts, >> >> >> With ovn-k8s, we need to keep the flows always on br-int which needed by >> running pods on the k8s node. >> >> Is there an ongoing project to address this problem? >> >> If not, I have one proposal not sure if it is doable. >> >> Please share your thoughts. >> The issue: >> >> In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on >> every K8s node. When we restart ovn-controller for upgrade using >> `ovs-appctl -t ovn-controller exit --restart`, the remaining traffic still >> works fine since br-int with flows still be Installed. >> >> >> >> However, when a new ovn-controller starts it will connect OVS IDL and do >> an engine init run, clearing all OpenFlow flows and install flows based on >> SB DB. >> >> With open flows count above 200K+, it took more than 15 seconds to get >> all the flows installed br-int bridge again. >> >> >> Proposal solution for the issue: >> >> When the ovn-controller gets “exit --start”, it will write a >> “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in >> external-ids column. When new ovn-controller starts, it will check if the >> “ovs-cond-seqno” exists in the Open_vSwitch table, and get the seqno from >> OVS IDL to decide if it will force a recomputing process? >> >> >> >> >> >> Hi Winson, >> >> >> >> Thanks for the proposal. Yes, the connection break during upgrading is a >> real issue in a large scale environment. However, the proposal doesn't >> work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB, >> which is a completely different connection from the ovs-vswitchd open-flow >> connection. >> >> To avoid clearing the open-flow table during ovn-controller startup, we >> can find a way to postpone clearing the OVS flows after the recomputing in >> ovn-controller is completed, right before ovn-controller replacing with the >> new flows. >> >> *[vi> ] * >> >> *[vi> ] Seems like we force recompute today if the OVS IDL is >> reconnected. Would it be possible to defer * >> >> *decision to recompute the flows based on the SB’s nb_cfg we have >> sync’d with? i.e. If our nb_cfg is * >> >> *in sync with the SB’s global nb_cfg, we can skip the recompute? At >> least if nothing has changed since* >> >> *the restart, we won’t need to do anything.. We could stash nb_cfg in OVS >> (once ovn-controller receives* >> >> *conformation from OVS that the physical flows for an nb_cfg update are >> in place), which should be cleared if * >> >> *OVS itself is restarted.. (I mean currently, nb_cfg is used to check if >> NB, SB and Chassis are in sync, we * >> >> *could extend this to OVS/physical flows?)* >> > > nb_cfg is already used by ovn-controller to do that, with the help of > "barrier" of OpenFlow, but I am not sure if it 100% working as expected. > > This basic idea should work, but in practice we need to take care of > generating the "installed" flow table and "desired" flow table in > ovn-controller. > I'd start with "postpone clearing OVS flows" which seems a lower hanging > fruit, and then see if any further improvement is needed. > > (resend using my gmail so that it can reach the ovn-kubernetes group.)
I thought about it again and it seems the idea of remembering nb_cfg doesn't work for the upgrading scenario. Even if nb_cfg is the same and we are sure about the flow that's installed in OVS reflects the certain nb_cfg version, we cannot say the OVS flows doesn't need any change, because the new version of ovn-controller implementation may translate same SB data into different OVS flows. So clearing the flow table is still the right thing to do, in terms of upgrading. (syncing back OVS flows from ovs-vswitchd to ovn-controller could avoid clearing the whole table, but that's a different approach as mentioned by Numan, and nb_cfg is not helpful anyway) Thanks, Han > >> >> *Have not thought through this though .. so maybe I am missing something…* >> >> >> >> *Thanks,* >> >> >> >> *-venu* >> >> This should largely reduce the time of connection broken during >> upgrading. Some changes in the ofctrl module's state machine are required, >> but I am not 100% sure if this approach is applicable. Need to check more >> details. >> >> >> >> Thanks, >> >> Han >> >> Test log: >> >> Check flow cnt on br-int every second: >> >> >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=0 >> >> packet_count=0 byte_count=0 flow_count=10322 >> >> packet_count=0 byte_count=0 flow_count=34220 >> >> packet_count=0 byte_count=0 flow_count=60425 >> >> packet_count=0 byte_count=0 flow_count=82506 >> >> packet_count=0 byte_count=0 flow_count=106771 >> >> packet_count=0 byte_count=0 flow_count=131648 >> >> packet_count=2 byte_count=120 flow_count=158303 >> >> packet_count=29 byte_count=1693 flow_count=185999 >> >> packet_count=188 byte_count=12455 flow_count=212764 >> >> >> >> >> >> -- >> >> Winson >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ovn-kubernetes" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to ovn-kubernetes+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com >> <https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ovn-kubernetes" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to ovn-kubernetes+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
