The numbers in the discussion you reference is for the CER. You are asking about and have CES deployed.
While the CES/CER are the same software platform, and probably more alike in hardware than anyone at Brocade will admit, they are different products marketed and sold with different feature sets and capabilities. The CER is primarily a router that does some switching. The CES is a layer 2 switch that also does some routing. I don't know the numbers, but the CES total routes in FIB is a significantly lower number than the CER. You might be able to order a software key "unlocking" a higher number. -----Original Message----- From: foundry-nsp [mailto:[email protected]] On Behalf Of Tamihiro Yuzawa Sent: Tuesday, August 20, 2013 7:18 PM To: [email protected] Subject: [f-nsp] CES FIB capacity revisited Helllo experts, especially Mr. Greg Hankins! A datacenter guy in Japan newly joining the list, please correct me in case whatever I'm saying is even slightly inappropriate here. I'm following this informative thread: https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html because there is a particular VRRP pair of CES2048CX boxes on our site that have issues with making FIB entries. > Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of CAM, > so for the CES/CER/CER-RT platforms the scalability is dynamic for all > IP routes in the FIB (vs the MLX CAM architecture where we have to > choose fixed partition sizes which can support a maximum number). Thank you Greg for the information. We had a Brocade Japan field engineer visiting our office yesterday, and he didn't realize CES would use SRAM for FIB instead of CAM. > Scalability depends on how the memory is used by IPv4 and IPv6 routes. > There is no easy formula to calculate the utilization, so we have > tested certain combinations which are officially supported maximums. > While a different number of routes might work, it will not be a > supported or tested combination. Very understandable. > "Internet route mix" is used to indicate that we are using a mix of > prefix lengths. No compression is being used. Understood. And here is our situation. Since the 3rd of August, we have started seeing a "Warning: IPv4 Network Route ADD: CAM entry creation FAILED" message every time we try to add a static IPv4 route, and a "show ip route detail" command for the prefix returns the following: > U_flags Entry_flags Age Cam:Index HW_Path_count > 0000e000 0 INVALID CAM 1 > > CAM Entry Flag: 00000000H > No cam index As such, the routers fail to forward packets destined to this prefix towards the corrrect NH. They just send packets to the deafault NH. (While it seems misleading they keep saying CAM when it's actually RAM, I don't mean to argue with that now :P) However, the number of routes they have is only <1200 (IPv4) and <500 (IPv6), which is far below the "officially supported maximum" that Greg kindly shared in his previous email. Here is a routing protocol breakdown of IPv4 routes in case it matters: D: 21 O: 860 O2: 74 S: 213 I should also mention that they are mostly /27 or longer prefix lengths. pref route len count ===== ===== 0: 2 16: 2 17: 2 18: 2 19: 0 20: 8 21: 2 22: 6 23: 80 24: 22 25: 14 26: 45 27: 134 28: 521 29: 85 30: 95 31: 24 32: 124 What's more is that except the two default (/0) and a few others, they are all subnets of either of two /21 prefixes. So I would say our routes are a far from evenly distributed mix of prefixes which is presumably unlike the "Internet route mix" that Greg used for testing. I assume if FIB in SRAM is stored in a tree structure, how many routes it can hold pretty much depends on the combination of what kind of prefix mix will be injected and how it's designed to store them. And my guess is that in our case, the aforementioned unevenness might be hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf split constraints, or something beyond my imagination. I'd appreciate insights from experts, hints, or suggestions for further investigations because this is becoming a serous issue since it's happening on our production PE routers. Thank you, -tami _______________________________________________ foundry-nsp mailing list [email protected] http://puck.nether.net/mailman/listinfo/foundry-nsp _______________________________________________ foundry-nsp mailing list [email protected] http://puck.nether.net/mailman/listinfo/foundry-nsp
