On Mon, Oct 24, 2005 at 10:43:58PM +0200, Gunther Stammwitz wrote: > Hello Brent, > > Thank you very much for your reply. This was a help to me. > #show ip cache > Total number of cache entries: 7529 > D:Dynamic P:Permanent F:Forward U:Us C:Complex Filter > W:Wait ARP I:ICMP Deny K:Drop R:Fragment S:Snap Encap > IP Address Next Hop MAC Type Port Vlan > Pri > 1 205.242.18.180 x.x.x.x 00d0.02e5.1bfc DF 2/4 491 > 0 > 2 62.96.30.11 x.x.x.x 0012.1e05.58db DF 1/1 492 > 0 > 3 213.247.32.99 DIRECT 0000.0000.0000 PU n/a > 0 > [..] > 1349 80.255.255.255 DIRECT 0000.0000.0000 PU n/a > 0 > 1350 255.255.255.255 DIRECT 0000.0000.0000 PU n/a > 0 > The ip-cache currently shows 7529 entries but when going through the full > list it ends at entry number 1350. Interesting... Maybe the entries are > timing out that fast? I have seen that one can set the timeout by "ip cache > age-time / DECIMAL amount of time before idle ip cache is age out" > but the online help doesn't list the format in which the values is given. > Maybe seconds? > It would also be interesting to see what the default value is. Any idea how > to find out? > > > The size of the cache itself seems to be configurable too: > system-max ip-cache > DECIMAL Valid range 8000 to 400000 (default: 140000) > > > But I guess the interesting thing is how many entries can be kept directly > in the cam as you say and doesn't need to be paged to the dram. > On an older B8GMR3-A-module the size seems to be limited to 512KB per DMA. > SL 1: B8GMR Fiber Management Module, SYSIF 2, M3, ACTIVE > Serial #: xxxx > 4096 KB BRAM, SMC version 1, BM version 21 > 512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 0, version 0209 > 512 KB PRAM(512K+0K) and shared CAM entries for DMA 1, version 0209 > 512 KB PRAM(512K+0K) and 2048*8 CAM entries for DMA 2, version 0209 > 512 KB PRAM(512K+0K) and shared CAM entries for DMA 3, version 0209 > > On a newer Jetcore module this size has been designed much larger... > SL 1: J-Bx2GMR4 JetCore Management Module, SYSIF 2 (Mini GBIC), M4, ACTIVE > Serial #: xxxx > 4096 KB BRAM, JetCore ASIC IGC version 49, BIA version 8a > 32768 KB PRAM and 2M-Bit*1 CAM for IGC 0, version 0449 > > > The interesting part seems to be how the cam is being divided up between > layer2,3 and 4 and how much there is left: > #show cam-partition brief > ==== SLOT 1 CAM PARTITION ==== > DMA: 0 (0x00) > Number of CAM devices per DMA: 8 > Number of hw entries per CAM: 0x00800 > Total size of CAM = 1Mbits > complete CAM index range per DMA: > (sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff) > (hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000) > Percentage of CAM hardware entries for each partition: > Layer3 = 12287 (0.749938Mbits) (74.993896%) > Level3 = 2047 (0.124938Mbits) (12.493896%) > Level2 = 2048 (0.125Mbits) (12.5%) > Layer4 Pool0 = 2048 (0.125Mbits) (12.5%) > Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%) > DMA: 2 (0x02) > Number of CAM devices per DMA: 8 > Number of hw entries per CAM: 0x00800 > Total size of CAM = 1Mbits > complete CAM index range per DMA: > (sw) 1 - 16383 (1 - 0x03fff), total entries: 16383 (0x03fff) > (hw) 0 - 16383 (0 - 0x03fff), total entries: 16384 (0x04000) > Percentage of CAM hardware entries for each partition: > Layer3 = 12287 (0.749938Mbits) (74.993896%) > Level3 = 2047 (0.124938Mbits) (12.493896%) > Level2 = 2048 (0.125Mbits) (12.5%) > Layer4 Pool0 = 2048 (0.125Mbits) (12.5%) > Layer2/Layer4 Pool1,2,3 = 2048 (0.125Mbits) (12.5%) > > AS one can see there are 16383 entries on the B8GMR3-A. To me it looks like > the system is creating an ip-cache-entry for every destination-ip of the > router. The whole Internet currently has about 170.000 entries so this is > the theoretical maximum but I guess there is no one who sends to *all* hosts > at the same time so this value seems to be sufficient if we keep in mind > that there is a timeout for the ip-cache. That's 170.000 _prefixes_. I don't think there are smaller prefixes than /24 out there today so with that in mind - yes you can easily fill your CAM, that is why the net-aggregate thing was invented. ip net-aggregate allows the CAM to save a cache entry for a whole net instead of for a single host.
> The number of cam entries on our router seems to be somewhere between 6000 > and 8000 over the time so we're at about 50% load but maybe someone from > foundry can explain how the mapping between DMAs, modules and Ethernet-ports > on those modules is being handled? > > Another friendly colleague told me to use the command "dm hist" which shows > if there are any overruns of the cam and luckily there aren't any. > > Let's see what happens if the next blaster-like worm appears and tries to > send traffic to a lot of different ips. Maybe one will hit the maximum cam > entries? I have. Actually your very email made me check our equipment and I found a box not having ip net-aggragate thus show ip cache: Total number of cache entries: 140000 net-aggregate is your friend! :) Kristian. > > > > One can also assume that game server-traffic where there's a constant > communication between servers and a whole bunch of clients is much more > stressing for a router since the ip-cache can't time out like with > http-requests where there is no keep-alive, right? > > > > What I have understood so far is that as long as routing is being done via > ip-cache and cam the cpu doesn't have to do anything with routing but my > question is still not completely answered: what is the limit of the > ip-forwarding-hardware? "Gigabit wire speed" can't be the answer since small > packets are much more stressing for a router than larger ones. Any ideas? > > Thanks to all of you who have answered so far :-) > > > Best regards, > Gunther > > > > > -----Urspr?ngliche Nachricht----- > > Von: Brent Van Dussen [mailto:vandusb at attens.com] > > Gesendet: Montag, 24. Oktober 2005 18:52 > > An: Gunther Stammwitz; foundry-nsp at puck.nether.net > > Betreff: Re: [f-nsp] Determining the load on the > > ip-forwarding engine on a Bigiron > > > > Hello Gunter, > > > > In terms of load we have always just had to keep an eye on > > the forwarding-cache size to get a gauge for when the switch > > is going to start having trouble. > > > > Since most of the forwarding is done in hardware at line-rate > > for all prefixes when net-aggregate is turned on, there > > usually isn't an issue. When you start to run into net-agg > > ineligibles, this necessitates the use of the CPU on the > > management card to keep track of and forward the non > > conforming packets. Since the CAM isn't big enough to carry > > all internet host's as separate entries, the CPU keeps track > > of them in a slower DRAM location that we call the ip-cache > > table ( 'show ip cache' ). > > > > So compare your current ip cache size with that of your > > maximum configured ip-cache size and you have a rough idea > > where you are at. Kind of an apples to Oranges comparison > > since a bigiron works nothing like a GSR does. > > > > I am not sure if CPU manipulation of the ip cache gets > > computed in with the "IP" section of show proc cpu, maybe a > > kind foundry developer can let us know :) > > > > -Brent > > > > At 11:30 AM 10/23/2005, Gunther Stammwitz wrote: > > >Hello colleagues, > > > > > >we're using several foundry Bigirons - some Ironcore and > > some Jetcore - > > >and I'd like to find out how much load there is on the > > ip-forwarding-engine. > > >On most Cisco routers one can check the cpu load of the > > Linecard but on > > >a foundry forwarding is done in hardware. Is there any way > > to find out > > >how much room there is left and how high the utilization is? > > > > > >As far as I understand a forwarding engine can usually > > forward x mpps > > >with a specific packet size which equals y megabits per > > second but much > > >less megabits if the packet size is smaller. > > > > > >Any ideas? > > >I've seen show proc cpu and one can read > > >"IP 0.19 0.13 0.13 0.15 > > 4518345" there > > >but I guess this means how much cpu of the management card is being > > >used by icmp-replies addressed to the router itself?!? > > > > > >Thanks, > > >Gunther > > >_______________________________________________ > > >foundry-nsp mailing list > > >foundry-nsp at puck.nether.net > > >http://puck.nether.net/mailman/listinfo/foundry-nsp > > > > > > > _______________________________________________ > foundry-nsp mailing list > foundry-nsp at puck.nether.net > http://puck.nether.net/mailman/listinfo/foundry-nsp
