On Tue, Nov 20, 2018 at 8:07 AM Mark Michelson <[email protected]> wrote: > > On 11/20/2018 02:13 AM, Han Zhou wrote: > > Hi Mark, thanks for writing up the interesting topic. I agree with Guru > > that holding packets and waiting for control plane change to get ready > > doesn't seem to work/scale. > > I am curious about the problem and I want to make sure I understand it > > correctly. It sounds like some FaaS scenario, is it? Is it reasonable in > > such scenario to keep at least one pod for the VIP even if it is idle? > > If not, is there any example system that actually works well by spinning > > up pods after requests came? > > This won't answer your questions directly, but I just met with OpenShift > networking devs and cleared something up. They currently have the > ability to spin up pods based on incoming packets to a VIP. However, > what they DON'T have is the ability to hold the incoming packet and > reinject it once pods are spun up. As it turns out, this is a new > feature request, rather than something that would put OVN in feature > parity with the current OpenShift SDN. >
I think there are two requirements here: 1. Trigger the pod spin up when packet comes. In theory this is doable as the options you described. However, I think it depends on how long the client can wait before it times out. What's the requirement for the e2e time? As Guru mentioned, it would be at least "seconds". How fast the pod spin-up finishes e2e, even without considering OVN? From OVN perspective, for the pods' network to be working, it depends on the scale and whether incremental processing is in place. It could be tens of seconds in large scale environment with current upstream code base. 2. Hold the packet and reinject. I wonder if this requirement is really necessary. L4 protocols usually should handle the first packet lost properly (just like how the first packet lost before ARP resolving is handled). For example, TCP SYN should retry and whether the first packets are buffered doesn't matter at all. Is there anything I am missing? Thanks, Han > > > > Regards, > > Han > > On Fri, Nov 16, 2018 at 11:28 AM Guru Shetty <[email protected] > > <mailto:[email protected]>> wrote: > > > > I don't have specific comments. But I do believe that "holding" > > packet may > > not work well as the time taken to create new pods, assign IPs to those > > pod, create new load-balancer rules for the new pods, will be in > > "seconds". > > > > > > On Fri, 9 Nov 2018 at 12:26, Mark Michelson <[email protected] > > <mailto:[email protected]>> wrote: > > > > > Yesterday during the OVN IRC meeting, I brought up a subject that > > > requires a bit more discussion. > > > > > > The issue: OpenShift has a use case where it uses a load > > balancer. The > > > VIP of the load balancer is backed with k8s pods. During times of > > > inactivity, these pods may be deleted due to idleness. > > Eventually, all > > > backing pods may be removed due to idleness. What OpenShift wants is > > > when a packet arrives for the load balancer VIP in this scenario, > > they > > > want to be able to react by creating new backing pods, updating load > > > balancer configuration, and reprocessing the packet so that it will > > > reach a destination pod. > > > > > > From an OVS/OVN perspective, what needs to happen is that based > > on an > > > incoming packet, an outside party needs to be notified to take an > > > action. The interesting part is at what layer the third party is > > > notified and how the incoming packet is further handled. We've > > discussed > > > this within Red Hat and have three ideas for how to do this. > > > > > > 1) The database approach > > > We add a new table to the southbound database, perhaps called > > > CMS_Events. In the situation where OpenShift wants unidling > > behavior, it > > > sets an option that results in ovn-controller installing a flow > > in OVS > > > to send the incoming packet to ovn-controller. ovn-controller, upon > > > receiving the incoming packet, buffers the packet and puts a new > > row in > > > the CMS_Events table. The CMS, upon seeing the event, takes its > > > necessary action. In this particular case, the CMS action is to > > spin up > > > pods and alter load balancer configuration. The CMS then sets a > > value in > > > row to indicate it has handled the event. ovn-controller, seeing the > > > event is handled, deletes the event and continues processing the > > packet, > > > allowing it to reach its destination. > > > > > > 2) The message passing approach > > > In the situation where OpenShift wants unidling behavior, it sets an > > > option that results in ovn-controller installing a flow in OVS to > > send > > > the incoming packet to ovn-controller. ovn-controller, upon receiving > > > the packet, buffers the packet and sends some sort of message to the > > > CMS. The contents of this message and the method by which the > > message is > > > sent would likely be dependent on the CMS in use. The CMS would > > act on > > > this message, spinning up new pods and altering load balancer > > > configuration. The CMS would then send a message back to > > ovn-controller, > > > where ovn-controller would continue processing the packet, > > allowing it > > > to reach its destination. > > > > > > 3) The second controller approach > > > In the situation where OpenShift wants unidling behavior, it sets an > > > option that results in ovn-controller installing a flow in OVS to > > send > > > the incoming packet to an external controller. The external > > controller, > > > upon receiving the packet, spins up new pods and alters load balancer > > > configuration. The external controller then (somehow) sends the > > packet > > > back into OVS so it can reach its destination. > > > > > > I presented the database approach during the meeting yesterday, > > but it's > > > worth presenting the other ideas we've discussed. Does any one of > > these > > > three ideas jump out as being significantly better than the > > others? Do > > > you have other ideas for how an external service can be contacted > > based > > > on the reception of a packet destined for a certain IP address? > > > > > > Thanks, > > > Mark Michelson > > > _______________________________________________ > > > dev mailing list > > > [email protected] <mailto:[email protected]> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > _______________________________________________ > > dev mailing list > > [email protected] <mailto:[email protected]> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
