Branch 1.10 Symptoms: 1. Flows corresponding to ICMP port unreachable errors cause EBADF messages in the log. 2. After a given time no more flows can be created.
There are 2 independent bugs: - Edouard has demonstrated with test code that if the vrouter determines that vr_flow_req_is_invalid() that the hold count variable that limits the number of flows in HOLD state “leaks”… i.e. when the flow is deleted the counter is not decremented. That means that after a sufficient number (4k) of “invalid” flow messages from the agent that no more flows can be created. - https://bugs.launchpad.net/juniperopenstack/+bug/1423848 has not been backported to 1.10. This is the root cause of the EBADF/invalid flow messages. The current fix for this bug is also incomplete in that it only considers ICMP type dest unreachable. It should use the same definition than the kernel for ICMP errors (vrouter/include/vr_packet.h:vr_icmp_error). It is important to include TIME_EXCEEDED as well as DEST_UNREACH. We need to update the fix and back port it to 1.10 branch. This bug triggers invalid flow messages in cases where the ICMP error reaches the vrouter before the flow entry is programmed. In the normal case, the sequence of events is (for an example UDP flow to an unreachable port): 1) UDP packet is sent 2) vrouter creates UDP flow HOLD entry 3) agent creates UDP forward and reverse flow entries in the vrouter. 4) ICMP error matches reverse flow in the vrouter table and is delivered. When the ICMP error arrives before the agent is able to program the flow entries in the kernel, then an additional HOLD entry is created in vrouter and a flow resolution request sent to the agent. If the agent has a different view of the flow key (due to missing/incomplete fix for 1423848) then it will generate a flow msg with an unexpected flow key, triggering EBADF/invalid flow msg errors. Since this is a timing issue it is difficult to reproduce accurately. Praveen, question: master does the following: PktHandler::ParseIpPacket() […] if ((pkt_info->agent_hdr.cmd == AgentHdr::TRAP_FLOW_MISS || pkt_info->agent_hdr.cmd == AgentHdr::TRAP_ECMP_RESOLVE) && (icmp->icmp_type == ICMP_DEST_UNREACH)) { ... } When back porting to 1.10 should the patch include the check for the command (agent_hdr.cmd) ? Plan: - Backport 1423848 to 1.10 (This should address the immediate cause of invalid flow messages). - Display the hold flow count on diagnostic commands (Anand). - Anand to investigate why the counter “leaks”. Pedro.
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
