On 21/10/16 10:23, Ewen McNeill wrote:
Does anyone have any suggestions for how to investigate what looks like
certain paths of inter-virtual-router layer 3 switching (routing, via
routes leaked into the VR) on a Juniper EX4600 (and I'd guess QFX5100)
being sent via the supervisor CPU?
For those playing along at home, it turns out that the Juniper EX
platform (apparently including the EX4600) does not support route
leaking of direct (connected) routes between virtual router instances:
https://kb.juniper.net/InfoCenter/index?page=content&id=KB23027&smlogin=true
This appears to manifest for us as:
- the direct routes are visible in the RIB (ie, from the supervisor CPU)
and "look normal" (eg, "show route table ...").
- the direct routes are not installed into the FIB (ie, for the packet
forwarding engine)
- normally in this case the traffic would follow some other (shorter
prefix) covering route (eg, default) instead, and perhaps take a round
about route; but in our case these VRs deliberately have no covering
route or default route
- the default behaviour if there is no route in the FIB is to reject the
packets
- which is done by queueing the packets to the supervisor CPU...
- and the supervisor CPU looks at its routing table (RIB) before
generating an ICMP Unreachable, finds a route in the RIB, so forwards
the traffic itself -- ie "process" switching, in this case supervisor
switching
As the KB article dryly observes "this software forwarding results in
heavy overhead on the CPU which causes high CPU utilization and can
cause unintended consequences". (I can't help thinking that maybe in
this case generating the ICMP Destination Unreachable anyway would have
made it more obvious that something was wrong, than software forwarding
the packet...)
The main work around is to ensure that all routes you leak on an EX
platform have a gateway router to send traffic to (rather than relying
on an ARP lookup) -- and if necessary to construct such an inter-VR
route with an additional linknet hop. (The KB article suggests a
firewall filter alternative, which presumably leverages other ASIC
features.)
Thanks to the very helpful Juniper NZ staff and Juniper JTAC staff for
assistance with investigating this, including JTAC for pointing us at
that KB article.
Since this configuration on my client's EX4600 is a transitional one
(separating one routing table into several, in advance of a new device
being installed which will route between the VRs eliminating the need
for leaked routes), we're going to redesign the transition plan to
ensure that there is a routed path with a gateway in both directions,
now as well rather than just later. (In our case the performance hit
was unidirectional as one direction happened to have a linknet to an
existing router, and the other direction had an interface that was the
default gateway for the connected subnet.)
Ewen
PS: I still do not know any way to recognise this "process switching"
happening other than observing the high latency (which grows rapidly
under any traffic load) symptoms, spotting the traffic in "monitor
traffic..." (which hints the supervisor CPU was involved) and then
looking for directly connected subnets that have been leaked between VRs.
_______________________________________________
NZNOG mailing list
[email protected]
https://list.waikato.ac.nz/mailman/listinfo/nznog
