Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
/me goes out to get Popcorn. :) Roger Harvey Mudd College On 3/16/2017 12:30 PM, Lee H Badman wrote: We need either a cage fight or a rock-off here pretty soon. *From:*The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] *On Behalf Of *Jeffrey D. Sessler *Sent:* Thursday, March 16, 2017 3:01 PM *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU *Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues There is a difference between interface speed and what the device can do. The 7240 was capable of 40Gbps only under a very specific large packet test (marketing talk). Under normal real-world traffic situations with a mix of packet sizes, the 7240 maxes at about 30% of the 40Gbps theoretical (CPU bound) the 8540 was 2x-3x better in the same real-world tests i.e. it can drive 6000 WAPs because it makes better use of its hardware. Jeff *From: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of "bosbo...@liberty.edu <mailto:bosbo...@liberty.edu>" mailto:bosbo...@liberty.edu>> *Reply-To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Date: *Thursday, March 16, 2017 at 11:25 AM *To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Both controllers are rated at 40 Gbps. I do not know about your place, but 40 Gps at up to 2040 APs make more sense than cramming 6000 APs into 40 Gps. * * *Bruce Osborne* */Senior Network Engineer/* *Network Operations - Wireless* * * * **(434) 592-4229* * * *LIBERTY UNIVERSITY* */Training Champions for Christ since 1971/* *From:*Jeffrey D. Sessler mailto:j...@scrippscollege.edu>> *Sent:* Thursday, March 16, 2017 1:52 PM *Subject:* Re: Cisco 8510 8.2 Load Issues So two 7240XM’s (to get almost the same performance) are less cost than a single 8540? :D Seriously, when you have both systems in production let’s talk. Remember, I have both here in my consortium, so I know a little bit about the costs and challenges of both. Jeff *From: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of "bosbo...@liberty.edu <mailto:bosbo...@liberty.edu>" mailto:bosbo...@liberty.edu>> *Reply-To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Date: *Thursday, March 16, 2017 at 10:15 AM *To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues You gain stateful firewall and lower cost. You also get vendor support so you do not need to depend on fellow users to determine stable code versions. * * *Bruce Osborne* */Senior Network Engineer/* *Network Operations - Wireless* * * * **(434) 592-4229* * * *LIBERTY UNIVERSITY* */Training Champions for Christ since 1971/* *From:*Jeffrey D. Sessler mailto:j...@scrippscollege.edu>> *Sent:* Thursday, March 16, 2017 11:14 AM *Subject:* Re: Cisco 8510 8.2 Load Issues But you’d need two of the 7240XM’s to almost equal the performance of a single 8540 :D Jeff *From: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of "bosbo...@liberty.edu <mailto:bosbo...@liberty.edu>" mailto:bosbo...@liberty.edu>> *Reply-To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Date: *Wednesday, March 15, 2017 at 11:03 PM *To: *"wireless-lan@listserv.educause.edu <mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> *Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Time for Aruba 7240XM, then. :D * * *Bruce Osborne* */Senior Network Engineer/* *Network Operations - Wireless* * * * **(434) 592-4229* * * *LIBERTY UNIVERSITY* */Training Champions for Christ since 1971/* *From:*Lee H Badman mailto:lhbad...@syr.edu>> *Sent:* Wednesday, March 15, 2017 1:22 PM *Subject:* Re: Cisco 8510 8.2 Load Issues Already on the 8540s. J *Lee Badman*| Network Archi
RE: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
We have a good mix of AP’s. that’s interesting to know. thanks -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jeffrey D. Sessler Sent: Friday, 10 March 2017 2:19 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues AVC – turn it off. If you’re using older model WAPs, AVC imparts a high cost on the controllers. On newer WAPs like the 3800, AVC is hardware accelerated. If you are going to run 8.2, MR5 is due any day now and you’ll want the new release. Jeff From: "wireless-lan@listserv.educause.edu<mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jake Snyder mailto:jsnyde...@gmail.com>> Reply-To: "wireless-lan@listserv.educause.edu<mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, March 8, 2017 at 10:13 PM To: "wireless-lan@listserv.educause.edu<mailto:wireless-lan@listserv.educause.edu>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone On Mar 8, 2017, at 10:56 PM, Jason Cook mailto:jason.c...@adelaide.edu.au>> wrote: We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migr
RE: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
We’ll certainly be asking those questions, though we don’t really use AVC at this point so.. happy to simply turn it off and have happy users ☺ -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Thursday, 9 March 2017 11:36 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues If you have 8510s with AVC issues, my advice to you is to go straight to the WBU and ask for real capabilities on this platform when AVC is running. Lee Badman | Network Architect Adjunct Instructor | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e lhbad...@syr.edu<mailto:lhbad...@syr.edu> w its.syr.edu SYRACUSE UNIVERSITY syr.edu From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, March 09, 2017 1:13 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone On Mar 8, 2017, at 10:56 PM, Jason Cook mailto:jason.c...@adelaide.edu.au>> wrote: We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migrating AP’s away to our old 5508’s, which saw
Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
AVC – turn it off. If you’re using older model WAPs, AVC imparts a high cost on the controllers. On newer WAPs like the 3800, AVC is hardware accelerated. If you are going to run 8.2, MR5 is due any day now and you’ll want the new release. Jeff From: "wireless-lan@listserv.educause.edu" on behalf of Jake Snyder Reply-To: "wireless-lan@listserv.educause.edu" Date: Wednesday, March 8, 2017 at 10:13 PM To: "wireless-lan@listserv.educause.edu" Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone On Mar 8, 2017, at 10:56 PM, Jason Cook mailto:jason.c...@adelaide.edu.au>> wrote: We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migrating AP’s away to our old 5508’s, which saw a significant drop just before 12:30 and things back to normal at 12:40 once 300AP’s were moved off. So for now users are happy, apparently we’ve even had callers in saying how good it is today (must have been bad the last week for that to happen). Controller response to SNMP was so bad it was taking Prime 2 minutes per AP to re-configure primary controller. Did it by hand, ssh/gui response was not it’s normal self but no problem. The 5508’s have shown no signs of being unhappy with about 150 AP’s each. We are working with TAC who have been good and they are investigating(no like cases found though), shedding the load has worked around the issue but it needs fixing. We upgraded to 8.2.141.0 yesterday eve
RE: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
If you have 8510s with AVC issues, my advice to you is to go straight to the WBU and ask for real capabilities on this platform when AVC is running. Lee Badman | Network Architect Adjunct Instructor | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e lhbad...@syr.edu<mailto:lhbad...@syr.edu> w its.syr.edu SYRACUSE UNIVERSITY syr.edu From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, March 09, 2017 1:13 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone On Mar 8, 2017, at 10:56 PM, Jason Cook mailto:jason.c...@adelaide.edu.au>> wrote: We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migrating AP’s away to our old 5508’s, which saw a significant drop just before 12:30 and things back to normal at 12:40 once 300AP’s were moved off. So for now users are happy, apparently we’ve even had callers in saying how good it is today (must have been bad the last week for that to happen). Controller response to SNMP was so bad it was taking Prime 2 minutes per AP to re-configure primary controller. Did it by hand, ssh/gui response was not it’s normal self but no problem. The 5508’s have shown no signs of being unhappy with about 150 AP’s each. We are working with TAC who have been good and
RE: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Fortunately we can see the issue starting before it really impacts users. So can 100% be sure of the issue we are going to put the load back on next Tuesday evening and continue to monitor and test Wednesday morning. At the first sign of issues disable AVC, and if no fix there migrate AP’s off again. It seems a good chance AVC is the issue. I hope so, then I can move on again I’ll report in next week ☺ Thanks again Jake -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:43 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone On Mar 8, 2017, at 10:56 PM, Jason Cook mailto:jason.c...@adelaide.edu.au>> wrote: We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migrating AP’s away to our old 5508’s, which saw a significant drop just before 12:30 and things back to normal at 12:40 once 300AP’s were moved off. So for now users are happy, apparently we’ve even had callers in saying how good it is today (must have been bad the last week for that to happen). Controller response to SNMP was so bad it was taking Prime 2 minutes per AP to re-configure primary controller. Did it by hand, ssh/gui response was not it’s normal self but no problem. The 5508’s have shown n
Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Might try leaving it off and see if that improves things. Just sounds oddly familiar. Make sure you disable it on all SSIDs to make sure you get a fair test with it off. Sent from my iPhone > On Mar 8, 2017, at 10:56 PM, Jason Cook wrote: > > We don’t use it, but yes looking at our SSID config under QOS AVC is enabled > on. Is this the only place to enable it? Per SSID? > > This would seem a good thing to kill off, clearly I should have paid more > attention to that discussion last year looking at history. > > Thanks Jake > > I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or > another > > -- > Jason Cook > Technology Services > The University of Adelaide, AUSTRALIA 5005 > Ph: +61 8 8313 4800 > > From: The EDUCAUSE Wireless Issues Constituent Group Listserv > [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder > Sent: Thursday, 9 March 2017 4:08 PM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues > > I hate to ask, but do you have AVC enabled? > > Sent from my iPhone > > On Mar 8, 2017, at 9:59 PM, Watters, John wrote: > > I'll check the load on our most loaded 8510 HA pair in the morning & get back > to you. It is about 2300-2500 APs with at least that many concurrent clients. > Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco > recommended that we move to 8.0.140 before going on up to 8.3). > > > > We just bought a new 8510HA pair for this same MPLS area to divide the load. > It is running, but has no load at all yet. Was thinking of starting it on 8.3 > code. So, I am very interested in your problem and tghe solution. Please be > sure to post it. > > > > > > > > > == > -jcw > From: The EDUCAUSE Wireless Issues Constituent Group Listserv > on behalf of Jason Cook > > Sent: Wednesday, March 8, 2017 10:28:09 PM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues > > Hi All, > > Just wondering if anyone has had an similar experiences to the fun we’ve had > the last week or so. > > Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we > had an issue in testing at the time so grabbed the latest ER release that > resolved a crash bug) > From 5x5508’s in N+1 on 8.0.121.0 code > We started before the end of term with a small number of locations but didn’t > fully load it up until the big break. Now the students are back and needing > there internet we have had some real load issues during the day. > > SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent > clients but the issue seems to creep in at about 10k. While ICMP isn’t the > greatest tool for performance it does line up here, the graph below show > around 10am we see increased delays in response to the vlan42 (client > network) interface on the controller and we see this on its management > interface too. At this point our clients ICMP to its own gateway starts to > increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows > 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and > even 0 at times. With users complaining of slowness and it’s worse unable to > login. > > CPU/Memory resources, channel util etc all ok. It’s site wide impact to users > no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So > seems in the controller itself. All testing done on 5hz > > Around midday we started migrating AP’s away to our old 5508’s, which saw a > significant drop just before 12:30 and things back to normal at 12:40 once > 300AP’s were moved off. So for now users are happy, apparently we’ve even had > callers in saying how good it is today (must have been bad the last week for > that to happen). Controller response to SNMP was so bad it was taking Prime 2 > minutes per AP to re-configure primary controller. Did it by hand, ssh/gui > response was not it’s normal self but no problem. The 5508’s have shown no > signs of being unhappy with about 150 AP’s each. > > We are working with TAC who have been good and they are investigating(no like > cases found though), shedding the load has worked around the issue but it > needs fixing. We upgraded to 8.2.141.0 yesterday evening but won’t be > re-loading the 8510’s until next week so confirming it’s fixed is a few days > off. There’s a few short upto 30ms delayed ICMP responses today but it’s hard > to know if that’s related or just the nature of icmp and network gear > priority. > > Interested to know if anyone
RE: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
We don’t use it, but yes looking at our SSID config under QOS AVC is enabled on. Is this the only place to enable it? Per SSID? This would seem a good thing to kill off, clearly I should have paid more attention to that discussion last year looking at history. Thanks Jake I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or another -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jake Snyder Sent: Thursday, 9 March 2017 4:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues I hate to ask, but do you have AVC enabled? Sent from my iPhone On Mar 8, 2017, at 9:59 PM, Watters, John mailto:john.watt...@ua.edu>> wrote: I'll check the load on our most loaded 8510 HA pair in the morning & get back to you. It is about 2300-2500 APs with at least that many concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco recommended that we move to 8.0.140 before going on up to 8.3). We just bought a new 8510HA pair for this same MPLS area to divide the load. It is running, but has no load at all yet. Was thinking of starting it on 8.3 code. So, I am very interested in your problem and tghe solution. Please be sure to post it. == -jcw From: The EDUCAUSE Wireless Issues Constituent Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Jason Cook mailto:jason.c...@adelaide.edu.au>> Sent: Wednesday, March 8, 2017 10:28:09 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues Hi All, Just wondering if anyone has had an similar experiences to the fun we’ve had the last week or so. Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we had an issue in testing at the time so grabbed the latest ER release that resolved a crash bug) From 5x5508’s in N+1 on 8.0.121.0 code We started before the end of term with a small number of locations but didn’t fully load it up until the big break. Now the students are back and needing there internet we have had some real load issues during the day. SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent clients but the issue seems to creep in at about 10k. While ICMP isn’t the greatest tool for performance it does line up here, the graph below show around 10am we see increased delays in response to the vlan42 (client network) interface on the controller and we see this on its management interface too. At this point our clients ICMP to its own gateway starts to increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and even 0 at times. With users complaining of slowness and it’s worse unable to login. CPU/Memory resources, channel util etc all ok. It’s site wide impact to users no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So seems in the controller itself. All testing done on 5hz Around midday we started migrating AP’s away to our old 5508’s, which saw a significant drop just before 12:30 and things back to normal at 12:40 once 300AP’s were moved off. So for now users are happy, apparently we’ve even had callers in saying how good it is today (must have been bad the last week for that to happen). Controller response to SNMP was so bad it was taking Prime 2 minutes per AP to re-configure primary controller. Did it by hand, ssh/gui response was not it’s normal self but no problem. The 5508’s have shown no signs of being unhappy with about 150 AP’s each. We are working with TAC who have been good and they are investigating(no like cases found though), shedding the load has worked around the issue but it needs fixing. We upgraded to 8.2.141.0 yesterday evening but won’t be re-loading the 8510’s until next week so confirming it’s fixed is a few days off. There’s a few short upto 30ms delayed ICMP responses today but it’s hard to know if that’s related or just the nature of icmp and network gear priority. Interested to know if anyone has seen anything like this in their environment. And anyone if anyone out there is using 8510’s in HA what’s your load in AP and concurrent users? I can imagine many places loading their devices up more than us Anyone know how to look at other hardware resources (not CPU/memory/system buffers) Something like ASIC on switches if it exists. Surely all this traffic isn’t cpu Thanks Jason -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph: +61 8 8313 4800 e-mail: jason.c...@adelaide.edu.au<mailto:jason.c...@adelaide.edu.au<mailto:jaso
Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
I hate to ask, but do you have AVC enabled? Sent from my iPhone > On Mar 8, 2017, at 9:59 PM, Watters, John wrote: > > I'll check the load on our most loaded 8510 HA pair in the morning & get back > to you. It is about 2300-2500 APs with at least that many concurrent clients. > Running 8.0.140.0 though (we moved there from a 7.6 (126 ?) level and Cisco > recommended that we move to 8.0.140 before going on up to 8.3). > > We just bought a new 8510HA pair for this same MPLS area to divide the load. > It is running, but has no load at all yet. Was thinking of starting it on 8.3 > code. So, I am very interested in your problem and tghe solution. Please be > sure to post it. > > > > > == > -jcw > From: The EDUCAUSE Wireless Issues Constituent Group Listserv > on behalf of Jason Cook > > Sent: Wednesday, March 8, 2017 10:28:09 PM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues > > Hi All, > > Just wondering if anyone has had an similar experiences to the fun we’ve had > the last week or so. > > Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11 (we > had an issue in testing at the time so grabbed the latest ER release that > resolved a crash bug) > From 5x5508’s in N+1 on 8.0.121.0 code > We started before the end of term with a small number of locations but didn’t > fully load it up until the big break. Now the students are back and needing > there internet we have had some real load issues during the day. > > SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent > clients but the issue seems to creep in at about 10k. While ICMP isn’t the > greatest tool for performance it does line up here, the graph below show > around 10am we see increased delays in response to the vlan42 (client > network) interface on the controller and we see this on its management > interface too. At this point our clients ICMP to its own gateway starts to > increase from 1-3ms to 400-600 and even upto 1800 when the big spike shows > 800ms to the interface. Iperf testing will also go from 100Mb down to 1-5 and > even 0 at times. With users complaining of slowness and it’s worse unable to > login. > > CPU/Memory resources, channel util etc all ok. It’s site wide impact to users > no matter if it’s HD rf design or what AP model (1142, 2702,3702,3502 etc) So > seems in the controller itself. All testing done on 5hz > > Around midday we started migrating AP’s away to our old 5508’s, which saw a > significant drop just before 12:30 and things back to normal at 12:40 once > 300AP’s were moved off. So for now users are happy, apparently we’ve even had > callers in saying how good it is today (must have been bad the last week for > that to happen). Controller response to SNMP was so bad it was taking Prime 2 > minutes per AP to re-configure primary controller. Did it by hand, ssh/gui > response was not it’s normal self but no problem. The 5508’s have shown no > signs of being unhappy with about 150 AP’s each. > > We are working with TAC who have been good and they are investigating(no like > cases found though), shedding the load has worked around the issue but it > needs fixing. We upgraded to 8.2.141.0 yesterday evening but won’t be > re-loading the 8510’s until next week so confirming it’s fixed is a few days > off. There’s a few short upto 30ms delayed ICMP responses today but it’s hard > to know if that’s related or just the nature of icmp and network gear > priority. > > Interested to know if anyone has seen anything like this in their environment. > And anyone if anyone out there is using 8510’s in HA what’s your load in AP > and concurrent users? I can imagine many places loading their devices up more > than us > Anyone know how to look at other hardware resources (not CPU/memory/system > buffers) Something like ASIC on switches if it exists. Surely all this > traffic isn’t cpu > > Thanks > > Jason > > > -- > Jason Cook > Technology Services > The University of Adelaide, AUSTRALIA 5005 > Ph: +61 8 8313 4800 > e-mail: jason.c...@adelaide.edu.au<mailto:jason.c...@adelaide.edu.au> > > CRICOS Provider Number 00123M > --- > This email message is intended only for the addressee(s) and contains > information which may be confidential and/or copyright. If you are not the > intended recipient please do not read, save, forward, disclose, or copy the > contents of this email. If this email has been sent to you in error, please > notify the sender by reply email and delete this email and
