Greg-san, thanks so much for your suggestion!
I have some interesting updates.
On Monday during the late hours, we evacuated some of the downlink customers to
another PE pair.
Even then, with only 730 IPv4 and 410 IPv6 routes, installing a single new /32
static route still failed.
Today, earlier in the morning we rebooted these boxes, one by one after driving
off the incoming traffic to the other, and brought the service traffic back
into the boxes.
And now, there's no problem with a new static route getting into FIB :)
U_flags Entry_flags Age Cam:Index HW_Path_count
0000e000 0 0002ce84 1
CAM Entry Flag: 00000001H
PPCR : 1:1 CIDX: 0002ce84 (IP_NETWORK: 0x2ce84)
#sh ip route sum
IP Routing Table - 731 entries
15 connected, 158 static, 0 RIP, 558 OSPF, 0 BGP, 0 ISIS
Number of prefixes:
/0: 1 /16: 1 /17: 1 /18: 1 /20: 4 /21: 1 /22: 3 /23: 40 /24: 19 /25: 7 /26:
23 /27: 68 /28: 275 /29: 74 /30: 92 /31: 24 /32: 97
Nexthop Table Entry - 161 entries
#sh ipv6 route sum
IPv6 Routing Table - 410 entries:
15 connected, 5 static, 0 RIP, 390 OSPF, 0 BGP, 0 ISIS
Number of prefixes:
/0:1 /32:1 /34:1 /64:386 /128:21
So what was it.. I mean before reboots?
Is it fair to suspect that memory fragmentation may have got too bad,
compaction work wasn't running as it should, and a reboot was the only way to
get some free space back into RAM?
Is there any way to get total SRAM size for FIB vs used amount via CLI?
We have already spoken to our VAR and requested that this issue be escalated to
Brocade TAC promptly.
I believe they will reach out for Brocade US engineers for further
investigation.
Greg-san, if you could follow up in any way, I'd much appreciate it.
We have deployed a bunch of CES pairs that took over FESX in our datacenters.
Therefore identifying the cause of this issue is very important. -tami
ghankins> Yuzawa-san, this really sounds like a technical issue that needs to be
ghankins> investigated by Brocade TAC and not by your VAR. The number of
routes,
ghankins> types of routes and the prefix distribution is so small that you
should
ghankins> not be seeing any FIB capacity issues on the CES.
ghankins>
ghankins> Kind regards,
ghankins> Greg
ghankins> (works for Brocade)
ghankins>
ghankins> --
ghankins> Greg Hankins <[email protected]>
ghankins>
ghankins> -----Original Message-----
ghankins> Date: Wed, 21 Aug 2013 11:18:20 +0900 (JST)
ghankins> From: Tamihiro Yuzawa <[email protected]>
ghankins> To: [email protected]
ghankins> Subject: [f-nsp] CES FIB capacity revisited
ghankins>
ghankins> Helllo experts, especially Mr. Greg Hankins!
ghankins>
ghankins> A datacenter guy in Japan newly joining the list, please correct me
in case whatever I'm saying is even slightly inappropriate here.
ghankins>
ghankins> I'm following this informative thread:
ghankins> https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html
ghankins>
ghankins> because there is a particular VRRP pair of CES2048CX boxes on our
site that have issues with making FIB entries.
ghankins>
ghankins> > Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of
CAM, so
ghankins> > for the CES/CER/CER-RT platforms the scalability is dynamic for all
IP
ghankins> > routes in the FIB (vs the MLX CAM architecture where we have to
choose
ghankins> > fixed partition sizes which can support a maximum number).
ghankins>
ghankins> Thank you Greg for the information.
ghankins> We had a Brocade Japan field engineer visiting our office yesterday,
and he didn't realize CES would use SRAM for FIB instead of CAM.
ghankins>
ghankins> > Scalability depends on how the memory is used by IPv4 and IPv6
routes.
ghankins> > There is no easy formula to calculate the utilization, so we have
tested
ghankins> > certain combinations which are officially supported maximums.
While a
ghankins> > different number of routes might work, it will not be a supported or
ghankins> > tested combination.
ghankins>
ghankins> Very understandable.
ghankins>
ghankins> > "Internet route mix" is used to indicate that we are using a mix of
prefix
ghankins> > lengths. No compression is being used.
ghankins>
ghankins> Understood.
ghankins>
ghankins> And here is our situation.
ghankins> Since the 3rd of August, we have started seeing a "Warning: IPv4
Network Route ADD: CAM entry creation FAILED" message every time we try to add
a static IPv4 route, and a "show ip route detail" command for the prefix
returns the following:
ghankins>
ghankins> > U_flags Entry_flags Age Cam:Index
HW_Path_count
ghankins> > 0000e000 0 INVALID CAM 1
ghankins> >
ghankins> > CAM Entry Flag: 00000000H
ghankins> > No cam index
ghankins>
ghankins> As such, the routers fail to forward packets destined to this prefix
towards the corrrect NH. They just send packets to the deafault NH.
ghankins> (While it seems misleading they keep saying CAM when it's actually
RAM, I don't mean to argue with that now :P)
ghankins>
ghankins> However, the number of routes they have is only <1200 (IPv4) and <500
(IPv6), which is far below the "officially supported maximum" that Greg kindly
shared in his previous email.
ghankins>
ghankins> Here is a routing protocol breakdown of IPv4 routes in case it
matters:
ghankins> D: 21
ghankins> O: 860
ghankins> O2: 74
ghankins> S: 213
ghankins>
ghankins> I should also mention that they are mostly /27 or longer prefix
lengths.
ghankins>
ghankins> pref route
ghankins> len count
ghankins> ===== =====
ghankins> 0: 2
ghankins> 16: 2
ghankins> 17: 2
ghankins> 18: 2
ghankins> 19: 0
ghankins> 20: 8
ghankins> 21: 2
ghankins> 22: 6
ghankins> 23: 80
ghankins> 24: 22
ghankins> 25: 14
ghankins> 26: 45
ghankins> 27: 134
ghankins> 28: 521
ghankins> 29: 85
ghankins> 30: 95
ghankins> 31: 24
ghankins> 32: 124
ghankins>
ghankins> What's more is that except the two default (/0) and a few others,
they are all subnets of either of two /21 prefixes.
ghankins> So I would say our routes are a far from evenly distributed mix of
prefixes which is presumably unlike the "Internet route mix" that Greg used for
testing.
ghankins>
ghankins> I assume if FIB in SRAM is stored in a tree structure, how many
routes it can hold pretty much depends on the combination of what kind of
prefix mix will be injected and how it's designed to store them.
ghankins>
ghankins> And my guess is that in our case, the aforementioned unevenness might
be hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf
split constraints, or something beyond my imagination.
ghankins>
ghankins> I'd appreciate insights from experts, hints, or suggestions for
further investigations because this is becoming a serous issue since it's
happening on our production PE routers. Thank you, -tami
_______________________________________________
foundry-nsp mailing list
[email protected]
http://puck.nether.net/mailman/listinfo/foundry-nsp
