> > > Hi Mark, thanks for writing up the interesting topic. I agree with Guru > > > that holding packets and waiting for control plane change to get ready > > > doesn't seem to work/scale. > > > I am curious about the problem and I want to make sure I understand it > > > correctly. It sounds like some FaaS scenario, is it? Is it reasonable in > > > such scenario to keep at least one pod for the VIP even if it is idle? > > > If not, is there any example system that actually works well by spinning > > > up pods after requests came?
Hi all, > > I think there are two requirements here: > > 1. Trigger the pod spin up when packet comes. > In theory this is doable as the options you described. However, I think it > depends on how long the client can wait before it times out. What's the > requirement for the e2e time? As Guru mentioned, it would be at least > "seconds". How fast the pod spin-up finishes e2e, even without considering > OVN? From OVN perspective, for the pods' network to be working, it depends > on the scale and whether incremental processing is in place. It could be > tens of seconds in large scale environment with current upstream code base. > I think point one is the basic requirement from OpenShift folks. As already described by Mark I guess the best option to establish a communication between ovn-controll and the CMS plugin is though a new table in the SB/NB DBs. A possible table implementation could be: - column 'event type' (e.g. type=unidling) - column data (generic info, e.g. L2/L3 addresses) - column done, used by the CMS to indicate that the event has been correctly served I think NB and SB tables can be mirrored. Are these info enough or something is missing? > 2. Hold the packet and reinject. > I wonder if this requirement is really necessary. L4 protocols usually > should handle the first packet lost properly (just like how the first > packet lost before ARP resolving is handled). For example, TCP SYN should > retry and whether the first packets are buffered doesn't matter at all. Is > there anything I am missing? > I guess this point is a 'nice to have' since the L4 timeout here will probably fix the issue Regards, Lorenzo > Thanks, > Han > > > > > > > Regards, > > > Han > > > On Fri, Nov 16, 2018 at 11:28 AM Guru Shetty <[email protected] > > > <mailto:[email protected]>> wrote: > > > > > > I don't have specific comments. But I do believe that "holding" > > > packet may > > > not work well as the time taken to create new pods, assign IPs to > those > > > pod, create new load-balancer rules for the new pods, will be in > > > "seconds". > > > > > > > > > On Fri, 9 Nov 2018 at 12:26, Mark Michelson <[email protected] > > > <mailto:[email protected]>> wrote: > > > > > > > Yesterday during the OVN IRC meeting, I brought up a subject that > > > > requires a bit more discussion. > > > > > > > > The issue: OpenShift has a use case where it uses a load > > > balancer. The > > > > VIP of the load balancer is backed with k8s pods. During times of > > > > inactivity, these pods may be deleted due to idleness. > > > Eventually, all > > > > backing pods may be removed due to idleness. What OpenShift > wants is > > > > when a packet arrives for the load balancer VIP in this scenario, > > > they > > > > want to be able to react by creating new backing pods, updating > load > > > > balancer configuration, and reprocessing the packet so that it > will > > > > reach a destination pod. > > > > > > > > From an OVS/OVN perspective, what needs to happen is that based > > > on an > > > > incoming packet, an outside party needs to be notified to take an > > > > action. The interesting part is at what layer the third party is > > > > notified and how the incoming packet is further handled. We've > > > discussed > > > > this within Red Hat and have three ideas for how to do this. > > > > > > > > 1) The database approach > > > > We add a new table to the southbound database, perhaps called > > > > CMS_Events. In the situation where OpenShift wants unidling > > > behavior, it > > > > sets an option that results in ovn-controller installing a flow > > > in OVS > > > > to send the incoming packet to ovn-controller. ovn-controller, > upon > > > > receiving the incoming packet, buffers the packet and puts a new > > > row in > > > > the CMS_Events table. The CMS, upon seeing the event, takes its > > > > necessary action. In this particular case, the CMS action is to > > > spin up > > > > pods and alter load balancer configuration. The CMS then sets a > > > value in > > > > row to indicate it has handled the event. ovn-controller, seeing > the > > > > event is handled, deletes the event and continues processing the > > > packet, > > > > allowing it to reach its destination. > > > > > > > > 2) The message passing approach > > > > In the situation where OpenShift wants unidling behavior, it > sets an > > > > option that results in ovn-controller installing a flow in OVS to > > > send > > > > the incoming packet to ovn-controller. ovn-controller, upon > receiving > > > > the packet, buffers the packet and sends some sort of message to > the > > > > CMS. The contents of this message and the method by which the > > > message is > > > > sent would likely be dependent on the CMS in use. The CMS would > > > act on > > > > this message, spinning up new pods and altering load balancer > > > > configuration. The CMS would then send a message back to > > > ovn-controller, > > > > where ovn-controller would continue processing the packet, > > > allowing it > > > > to reach its destination. > > > > > > > > 3) The second controller approach > > > > In the situation where OpenShift wants unidling behavior, it > sets an > > > > option that results in ovn-controller installing a flow in OVS to > > > send > > > > the incoming packet to an external controller. The external > > > controller, > > > > upon receiving the packet, spins up new pods and alters load > balancer > > > > configuration. The external controller then (somehow) sends the > > > packet > > > > back into OVS so it can reach its destination. > > > > > > > > I presented the database approach during the meeting yesterday, > > > but it's > > > > worth presenting the other ideas we've discussed. Does any one of > > > these > > > > three ideas jump out as being significantly better than the > > > others? Do > > > > you have other ideas for how an external service can be contacted > > > based > > > > on the reception of a packet destined for a certain IP address? > > > > > > > > Thanks, > > > > Mark Michelson > > > > _______________________________________________ > > > > dev mailing list > > > > [email protected] <mailto:[email protected]> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > _______________________________________________ > > > dev mailing list > > > [email protected] <mailto:[email protected]> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
