Hi David, thank you for the clarification. I should have referred to their datasheets as for the specific FIB size of the CES. http://www.brocade.com/downloads/documents/data_sheets/product_data_sheets/netiron-ces-2000-ds.pdf
Yes, compared to CER, FIB in CES is very small. Especially now the CER (without RT) has as much capacity as the former CER-RT. But still, they say the CES series can store... - Up to 32,000 IPv4 unicast routes in FIB - Up to 8000 IPv6 unicast routes in FIB Please take a look at another email that I posted on this thread about an hour ago. The problem has gone after a reboot :) Let's wait and see what their TAC has to say. Best regards, -tami dunfried> The numbers in the discussion you reference is for the CER. You are asking dunfried> about and have CES deployed. dunfried> dunfried> While the CES/CER are the same software platform, and probably more alike in dunfried> hardware than anyone at Brocade will admit, they are different products dunfried> marketed and sold with different feature sets and capabilities. The CER is dunfried> primarily a router that does some switching. The CES is a layer 2 switch dunfried> that also does some routing. dunfried> I don't know the numbers, but the CES total routes in FIB is a significantly dunfried> lower number than the CER. dunfried> You might be able to order a software key "unlocking" a higher number. dunfried> dunfried> -----Original Message----- dunfried> From: foundry-nsp [mailto:[email protected]] On Behalf Of dunfried> Tamihiro Yuzawa dunfried> Sent: Tuesday, August 20, 2013 7:18 PM dunfried> To: [email protected] dunfried> Subject: [f-nsp] CES FIB capacity revisited dunfried> dunfried> Helllo experts, especially Mr. Greg Hankins! dunfried> dunfried> A datacenter guy in Japan newly joining the list, please correct me in case dunfried> whatever I'm saying is even slightly inappropriate here. dunfried> dunfried> I'm following this informative thread: dunfried> https://puck.nether.net/pipermail/foundry-nsp/2012-October/003719.html dunfried> dunfried> because there is a particular VRRP pair of CES2048CX boxes on our site that dunfried> have issues with making FIB entries. dunfried> dunfried> > Hi Rob, the CER routers use SRAM for the IPv4/v6 FIB instead of CAM, dunfried> > so for the CES/CER/CER-RT platforms the scalability is dynamic for all dunfried> > IP routes in the FIB (vs the MLX CAM architecture where we have to dunfried> > choose fixed partition sizes which can support a maximum number). dunfried> dunfried> Thank you Greg for the information. dunfried> We had a Brocade Japan field engineer visiting our office yesterday, and he dunfried> didn't realize CES would use SRAM for FIB instead of CAM. dunfried> dunfried> > Scalability depends on how the memory is used by IPv4 and IPv6 routes. dunfried> > There is no easy formula to calculate the utilization, so we have dunfried> > tested certain combinations which are officially supported maximums. dunfried> > While a different number of routes might work, it will not be a dunfried> > supported or tested combination. dunfried> dunfried> Very understandable. dunfried> dunfried> > "Internet route mix" is used to indicate that we are using a mix of dunfried> > prefix lengths. No compression is being used. dunfried> dunfried> Understood. dunfried> dunfried> And here is our situation. dunfried> Since the 3rd of August, we have started seeing a "Warning: IPv4 Network dunfried> Route ADD: CAM entry creation FAILED" message every time we try to add a dunfried> static IPv4 route, and a "show ip route detail" command for the prefix dunfried> returns the following: dunfried> dunfried> > U_flags Entry_flags Age Cam:Index HW_Path_count dunfried> > 0000e000 0 INVALID CAM 1 dunfried> > dunfried> > CAM Entry Flag: 00000000H dunfried> > No cam index dunfried> dunfried> As such, the routers fail to forward packets destined to this prefix towards dunfried> the corrrect NH. They just send packets to the deafault NH. dunfried> (While it seems misleading they keep saying CAM when it's actually RAM, I dunfried> don't mean to argue with that now :P) dunfried> dunfried> However, the number of routes they have is only <1200 (IPv4) and <500 dunfried> (IPv6), which is far below the "officially supported maximum" that Greg dunfried> kindly shared in his previous email. dunfried> dunfried> Here is a routing protocol breakdown of IPv4 routes in case it matters: dunfried> D: 21 dunfried> O: 860 dunfried> O2: 74 dunfried> S: 213 dunfried> dunfried> I should also mention that they are mostly /27 or longer prefix lengths. dunfried> dunfried> pref route dunfried> len count dunfried> ===== ===== dunfried> 0: 2 dunfried> 16: 2 dunfried> 17: 2 dunfried> 18: 2 dunfried> 19: 0 dunfried> 20: 8 dunfried> 21: 2 dunfried> 22: 6 dunfried> 23: 80 dunfried> 24: 22 dunfried> 25: 14 dunfried> 26: 45 dunfried> 27: 134 dunfried> 28: 521 dunfried> 29: 85 dunfried> 30: 95 dunfried> 31: 24 dunfried> 32: 124 dunfried> dunfried> What's more is that except the two default (/0) and a few others, they are dunfried> all subnets of either of two /21 prefixes. dunfried> So I would say our routes are a far from evenly distributed mix of prefixes dunfried> which is presumably unlike the "Internet route mix" that Greg used for dunfried> testing. dunfried> dunfried> I assume if FIB in SRAM is stored in a tree structure, how many routes it dunfried> can hold pretty much depends on the combination of what kind of prefix mix dunfried> will be injected and how it's designed to store them. dunfried> dunfried> And my guess is that in our case, the aforementioned unevenness might be dunfried> hitting some scale limitations on FIB in CES, be it SRAM partitions, leaf dunfried> split constraints, or something beyond my imagination. dunfried> dunfried> I'd appreciate insights from experts, hints, or suggestions for further dunfried> investigations because this is becoming a serous issue since it's happening dunfried> on our production PE routers. Thank you, -tami dunfried> _______________________________________________ dunfried> foundry-nsp mailing list dunfried> [email protected] dunfried> http://puck.nether.net/mailman/listinfo/foundry-nsp dunfried> dunfried> _______________________________________________ foundry-nsp mailing list [email protected] http://puck.nether.net/mailman/listinfo/foundry-nsp
