Re: [ovs-discuss] Flow miss/Packet order question
On Oct 3, 2013, at 22:41 PM, Jesse Gross je...@nicira.com wrote: On Thu, Oct 3, 2013 at 6:26 AM, Dmitry Fleytman dfley...@redhat.com wrote: On Oct 3, 2013, at 04:20 AM, Jesse Gross je...@nicira.com wrote: On Wed, Oct 2, 2013 at 4:49 AM, Dmitry Fleytman dfley...@redhat.com wrote: On Apr 30, 2012, at 20:15 PM, Ben Pfaff b...@nicira.com wrote: I think that your explanation stems from a misunderstanding. Yes, if an OpenFlow controller uses a reactive model, then it cannot avoid the problem. However, I think that Joji is raising a different issue, one that is an implementation detail within Open vSwitch and that controllers have no power to avoid. Let me explain in detail. When a packet arrives for which there is no kernel flow, the kernel sends it to userspace. Userspace sends the packet and sets up a kernel flow. In the meantime, more packets might have arrived and been queued to userspace. Userspace will send these packets, but any packets that arrive after the kernel flow is set up will be forwarded directly by the kernel before those queued to userspace go out. This is exactly the problem we face while going for KVM paravirtualized network driver for Windows (NetKVM) certification. There are a few automated tests that send bursts of packets and wait for the same packets and the same order on the other side. We have a POC patches (pretty dirty) that solve the problem (below). The idea is simple - when datapath makes upcall it queues packets in kernel until user mode completes processing and downloads a new flow. It looks like overkill to queue packets per datapath, queueing per vport will be enough, but it was easier to implement this way and it proves the concept as well. Still, it is obvious there is performance and scaling impact so another ideas are highly welcome. What do you think? Should we go for this solution and prepare clean patches for submission? I think in order to fully solve the problem you actually need to queue per flow, rather than per port or per datapath. Otherwise, you end up serializing to one flow setup at a time, which is probably a bigger problem in practice than the one this is trying to solve. The problem is we are talking about moments when there is no flow for specific packet and user mode is working on it's creation. How can we serialise per flow in this case? You would have to track pending flows in some form, either through the flow table or a new data structure and release packets when a real flow is installed. I hope that there is a simpler way of solving this problem because it would add quite a bit of complexity (not just managing all of these queues but also enforcing limits like a socket buffer). However, this is no way that serializing at a per datapath or per port level is going to perform well enough to make that a viable solution. Ok, got it. Thanks for the idea. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
I've known about this problem for years now, but I've always ignored it because it didn't seem to cause trouble in practice. I have a favorite straw-man approach. It won't work as-is, but perhaps it will spur someone to an idea that would work. Suppose that we changed the semantics of the kernel flow_put so that, when the kernel puts the flow, it locks the socket buffer of the Netlink socket used to send packets in that flow to userspace, scans it for other packets in the flow, removes them from the socket buffer, and executes their actions. Problems with this idea include, at least: 1. Nothing does anything like this with socket buffers (that is, there is no precedent for deleting packets queued to userspace from a socket buffer) and it seems wrong. 2. Identifying the socket buffer or buffers to scan is problematic (maybe the client would have to identify them). 3. The flow_put would have to arrange for any other packets arriving after it locks the socket buffer to block until it finishes processing the buffered packets, so that those arriving packets would be processed through the new flow in the normal way, but guaranteed to be ordered after the ones in the socket buffer. (This probably isn't hard.) ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
On Thu, Oct 3, 2013 at 6:26 AM, Dmitry Fleytman dfley...@redhat.com wrote: On Oct 3, 2013, at 04:20 AM, Jesse Gross je...@nicira.com wrote: On Wed, Oct 2, 2013 at 4:49 AM, Dmitry Fleytman dfley...@redhat.com wrote: On Apr 30, 2012, at 20:15 PM, Ben Pfaff b...@nicira.com wrote: I think that your explanation stems from a misunderstanding. Yes, if an OpenFlow controller uses a reactive model, then it cannot avoid the problem. However, I think that Joji is raising a different issue, one that is an implementation detail within Open vSwitch and that controllers have no power to avoid. Let me explain in detail. When a packet arrives for which there is no kernel flow, the kernel sends it to userspace. Userspace sends the packet and sets up a kernel flow. In the meantime, more packets might have arrived and been queued to userspace. Userspace will send these packets, but any packets that arrive after the kernel flow is set up will be forwarded directly by the kernel before those queued to userspace go out. This is exactly the problem we face while going for KVM paravirtualized network driver for Windows (NetKVM) certification. There are a few automated tests that send bursts of packets and wait for the same packets and the same order on the other side. We have a POC patches (pretty dirty) that solve the problem (below). The idea is simple - when datapath makes upcall it queues packets in kernel until user mode completes processing and downloads a new flow. It looks like overkill to queue packets per datapath, queueing per vport will be enough, but it was easier to implement this way and it proves the concept as well. Still, it is obvious there is performance and scaling impact so another ideas are highly welcome. What do you think? Should we go for this solution and prepare clean patches for submission? I think in order to fully solve the problem you actually need to queue per flow, rather than per port or per datapath. Otherwise, you end up serializing to one flow setup at a time, which is probably a bigger problem in practice than the one this is trying to solve. The problem is we are talking about moments when there is no flow for specific packet and user mode is working on it's creation. How can we serialise per flow in this case? You would have to track pending flows in some form, either through the flow table or a new data structure and release packets when a real flow is installed. I hope that there is a simpler way of solving this problem because it would add quite a bit of complexity (not just managing all of these queues but also enforcing limits like a socket buffer). However, this is no way that serializing at a per datapath or per port level is going to perform well enough to make that a viable solution. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
On Wed, Oct 2, 2013 at 4:49 AM, Dmitry Fleytman dfley...@redhat.com wrote: On Apr 30, 2012, at 20:15 PM, Ben Pfaff b...@nicira.com wrote: I think that your explanation stems from a misunderstanding. Yes, if an OpenFlow controller uses a reactive model, then it cannot avoid the problem. However, I think that Joji is raising a different issue, one that is an implementation detail within Open vSwitch and that controllers have no power to avoid. Let me explain in detail. When a packet arrives for which there is no kernel flow, the kernel sends it to userspace. Userspace sends the packet and sets up a kernel flow. In the meantime, more packets might have arrived and been queued to userspace. Userspace will send these packets, but any packets that arrive after the kernel flow is set up will be forwarded directly by the kernel before those queued to userspace go out. This is exactly the problem we face while going for KVM paravirtualized network driver for Windows (NetKVM) certification. There are a few automated tests that send bursts of packets and wait for the same packets and the same order on the other side. We have a POC patches (pretty dirty) that solve the problem (below). The idea is simple - when datapath makes upcall it queues packets in kernel until user mode completes processing and downloads a new flow. It looks like overkill to queue packets per datapath, queueing per vport will be enough, but it was easier to implement this way and it proves the concept as well. Still, it is obvious there is performance and scaling impact so another ideas are highly welcome. What do you think? Should we go for this solution and prepare clean patches for submission? I think in order to fully solve the problem you actually need to queue per flow, rather than per port or per datapath. Otherwise, you end up serializing to one flow setup at a time, which is probably a bigger problem in practice than the one this is trying to solve. It's not entirely clear to me that the general solution is really worth the extra complexity for situations beyond the WHQL test so there might be a bandaid for that particular problem. Do you know exactly what it is doing? ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
I currently do not have a real scenario where I have run into this problem. However, it is easy to see real scenarios where I could run into this. As you said, most flows start off with a single packet and wait for a response. But, there is also the flow eviction mechanism which would bring this into play mid-flow. There is also the UDP traffic which does not have a ramp up mechanism. Yes, the window is small. But, I think it is real and could problems for some applications. -Joji On Thu, Apr 26, 2012 at 10:22 PM, Ben Pfaff b...@nicira.com wrote: It isn't commonly a problem in practice because flows most often start off with a single packet and wait for a return packet before ramping up packet volume. I've been aware of the issue for years, but you are the only other person I ever recall bringing it up on the mailing lists. Do you have a real situation (not a hypothetical scenario) where you see this causing trouble? On Fri, Apr 27, 2012 at 11:06:29AM +0600, junaid khalid wrote: Are you planning to solve this problem in near future or do you have any suggestions to mitigate this problem? On Thu, Apr 26, 2012 at 2:37 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 01:33:56PM -0700, Joji Mtt wrote: I am trying to figure out if there would be a packet order issue with the current version of OVS. Consider a case where a controller has added a forwarding rule for a specific flow (Flow A) and this rule is not yet installed in the DP. In this scenario, it is conceivable that certain (bursty) traffic patterns on Flow A can result in the packets being sent out of order. E.g. consider an initial burst of 5 packets that miss the kernel flow table, followed by several subsequent packets arriving at random intervals. As soon as the userspace processes the flow miss, it will install a rule in the kernel. Depending on the relative timing of the rule installation, any of these subsequent packets could get switched directly by the kernel before the previous packets that took the slow path could get forwarded. I couldn't find any special handling to cover this case. Most likely it is already handled and I am just missing the part where it is done. Could anyone clarify this for me? Yes, it's possible to get out-of-order packets for this reason. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
It does indeed happen with UDP flows (or if a rule is removed while a flow is active) and it's easy to observe. It's not a bug in Open vSwitch, which is in fact implementing the correct OpenFlow behavior. In practice you can easily solve the problem either by pushing down routes proactively or by buffering somewhere (e.g. at the receiver.) On Apr 27, 2012, at 7:39 AM, Joji Mtt wrote: I currently do not have a real scenario where I have run into this problem. However, it is easy to see real scenarios where I could run into this. As you said, most flows start off with a single packet and wait for a response. But, there is also the flow eviction mechanism which would bring this into play mid-flow. There is also the UDP traffic which does not have a ramp up mechanism. Yes, the window is small. But, I think it is real and could problems for some applications. -Joji On Thu, Apr 26, 2012 at 10:22 PM, Ben Pfaff b...@nicira.com wrote: It isn't commonly a problem in practice because flows most often start off with a single packet and wait for a return packet before ramping up packet volume. I've been aware of the issue for years, but you are the only other person I ever recall bringing it up on the mailing lists. Do you have a real situation (not a hypothetical scenario) where you see this causing trouble? On Fri, Apr 27, 2012 at 11:06:29AM +0600, junaid khalid wrote: Are you planning to solve this problem in near future or do you have any suggestions to mitigate this problem? On Thu, Apr 26, 2012 at 2:37 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 01:33:56PM -0700, Joji Mtt wrote: I am trying to figure out if there would be a packet order issue with the current version of OVS. Consider a case where a controller has added a forwarding rule for a specific flow (Flow A) and this rule is not yet installed in the DP. In this scenario, it is conceivable that certain (bursty) traffic patterns on Flow A can result in the packets being sent out of order. E.g. consider an initial burst of 5 packets that miss the kernel flow table, followed by several subsequent packets arriving at random intervals. As soon as the userspace processes the flow miss, it will install a rule in the kernel. Depending on the relative timing of the rule installation, any of these subsequent packets could get switched directly by the kernel before the previous packets that took the slow path could get forwarded. I couldn't find any special handling to cover this case. Most likely it is already handled and I am just missing the part where it is done. Could anyone clarify this for me? Yes, it's possible to get out-of-order packets for this reason. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
Are you planning to solve this problem in near future or do you have any suggestions to mitigate this problem? On Thu, Apr 26, 2012 at 2:37 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 01:33:56PM -0700, Joji Mtt wrote: I am trying to figure out if there would be a packet order issue with the current version of OVS. Consider a case where a controller has added a forwarding rule for a specific flow (Flow A) and this rule is not yet installed in the DP. In this scenario, it is conceivable that certain (bursty) traffic patterns on Flow A can result in the packets being sent out of order. E.g. consider an initial burst of 5 packets that miss the kernel flow table, followed by several subsequent packets arriving at random intervals. As soon as the userspace processes the flow miss, it will install a rule in the kernel. Depending on the relative timing of the rule installation, any of these subsequent packets could get switched directly by the kernel before the previous packets that took the slow path could get forwarded. I couldn't find any special handling to cover this case. Most likely it is already handled and I am just missing the part where it is done. Could anyone clarify this for me? Yes, it's possible to get out-of-order packets for this reason. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
Re: [ovs-discuss] Flow miss/Packet order question
It isn't commonly a problem in practice because flows most often start off with a single packet and wait for a return packet before ramping up packet volume. I've been aware of the issue for years, but you are the only other person I ever recall bringing it up on the mailing lists. Do you have a real situation (not a hypothetical scenario) where you see this causing trouble? On Fri, Apr 27, 2012 at 11:06:29AM +0600, junaid khalid wrote: Are you planning to solve this problem in near future or do you have any suggestions to mitigate this problem? On Thu, Apr 26, 2012 at 2:37 AM, Ben Pfaff b...@nicira.com wrote: On Wed, Apr 25, 2012 at 01:33:56PM -0700, Joji Mtt wrote: I am trying to figure out if there would be a packet order issue with the current version of OVS. Consider a case where a controller has added a forwarding rule for a specific flow (Flow A) and this rule is not yet installed in the DP. In this scenario, it is conceivable that certain (bursty) traffic patterns on Flow A can result in the packets being sent out of order. E.g. consider an initial burst of 5 packets that miss the kernel flow table, followed by several subsequent packets arriving at random intervals. As soon as the userspace processes the flow miss, it will install a rule in the kernel. Depending on the relative timing of the rule installation, any of these subsequent packets could get switched directly by the kernel before the previous packets that took the slow path could get forwarded. I couldn't find any special handling to cover this case. Most likely it is already handled and I am just missing the part where it is done. Could anyone clarify this for me? Yes, it's possible to get out-of-order packets for this reason. ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
[ovs-discuss] Flow miss/Packet order question
Hi- I am trying to figure out if there would be a packet order issue with the current version of OVS. Consider a case where a controller has added a forwarding rule for a specific flow (Flow A) and this rule is not yet installed in the DP. In this scenario, it is conceivable that certain (bursty) traffic patterns on Flow A can result in the packets being sent out of order. E.g. consider an initial burst of 5 packets that miss the kernel flow table, followed by several subsequent packets arriving at random intervals. As soon as the userspace processes the flow miss, it will install a rule in the kernel. Depending on the relative timing of the rule installation, any of these subsequent packets could get switched directly by the kernel before the previous packets that took the slow path could get forwarded. I couldn't find any special handling to cover this case. Most likely it is already handled and I am just missing the part where it is done. Could anyone clarify this for me? Thanks, Joji ___ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
