We need either a cage fight or a rock-off here pretty soon.
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jeffrey D. Sessler
*Sent:* Thursday, March 16, 2017 3:01 PM
*To:* [email protected]
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
There is a difference between interface speed and what the device can do. The
7240 was capable of 40Gbps only under a very specific large packet test
(marketing talk). Under normal real-world traffic situations with a mix of
packet sizes, the 7240 maxes at about 30% of the 40Gbps theoretical (CPU bound)
the 8540 was 2x-3x better in the same real-world tests i.e. it can drive 6000
WAPs because it makes better use of its hardware.
Jeff
*From: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>> on behalf of "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Reply-To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Date: *Thursday, March 16, 2017 at 11:25 AM
*To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Both controllers are rated at 40 Gbps.
I do not know about your place, but 40 Gps at up to 2040 APs make more sense
than cramming 6000 APs into 40 Gps.
* *
*Bruce Osborne*
*/Senior Network Engineer/*
*Network Operations - Wireless*
* *
* **(434) 592-4229*
* *
*LIBERTY UNIVERSITY*
*/Training Champions for Christ since 1971/*
--------------------------------------------------------------------------------
*From:*Jeffrey D. Sessler <[email protected]
<mailto:[email protected]>>
*Sent:* Thursday, March 16, 2017 1:52 PM
*Subject:* Re: Cisco 8510 8.2 Load Issues
So two 7240XM’s (to get almost the same performance) are less cost than a single
8540? :D
Seriously, when you have both systems in production let’s talk. Remember, I have
both here in my consortium, so I know a little bit about the costs and
challenges of both.
Jeff
*From: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>> on behalf of "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Reply-To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Date: *Thursday, March 16, 2017 at 10:15 AM
*To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
You gain stateful firewall and lower cost.
You also get vendor support so you do not need to depend on fellow users to
determine stable code versions.
* *
*Bruce Osborne*
*/Senior Network Engineer/*
*Network Operations - Wireless*
* *
* **(434) 592-4229*
* *
*LIBERTY UNIVERSITY*
*/Training Champions for Christ since 1971/*
--------------------------------------------------------------------------------
*From:*Jeffrey D. Sessler <[email protected]
<mailto:[email protected]>>
*Sent:* Thursday, March 16, 2017 11:14 AM
*Subject:* Re: Cisco 8510 8.2 Load Issues
But you’d need two of the 7240XM’s to almost equal the performance of a single
8540 :D
Jeff
*From: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>> on behalf of "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Reply-To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Date: *Wednesday, March 15, 2017 at 11:03 PM
*To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Time for Aruba 7240XM, then. :D
* *
*Bruce Osborne*
*/Senior Network Engineer/*
*Network Operations - Wireless*
* *
* **(434) 592-4229*
* *
*LIBERTY UNIVERSITY*
*/Training Champions for Christ since 1971/*
--------------------------------------------------------------------------------
*From:*Lee H Badman <[email protected] <mailto:[email protected]>>
*Sent:* Wednesday, March 15, 2017 1:22 PM
*Subject:* Re: Cisco 8510 8.2 Load Issues
Already on the 8540s. J
*Lee Badman*| Network Architect
Adjunct Instructor | CWNE #200
Information Technology Services
206 Machinery Hall
120 Smith Drive
Syracuse, New York 13244
*t*315.443.3003 *f* 315.443.4325 *e* [email protected]
<mailto:[email protected]> *w* its.syr.edu
*SYRACUSE UNIVERSITY*
syr.edu
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jeffrey D. Sessler
*Sent:* Wednesday, March 15, 2017 11:24 AM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Talk with your sales team. There are some amazing bundle discounts right now for
their end-of-year. One I saw added an additional 38% off when purchasing
controller/WAP/switches together. Could be a good opportunity to move to the
8540 controllers (which are beasts).
Jeff
*From: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>> on behalf of "[email protected]
<mailto:[email protected]>" <[email protected] <mailto:[email protected]>>
*Reply-To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Date: *Wednesday, March 15, 2017 at 5:15 AM
*To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Exactly what we saw a while back. It was devastating to all WLANs on the
controller, even with AVC on just a couple. As in catastrophic, and at the time
we needed AVC. I don’t **believe** it’s a code issue, but more a HW limit.
*Lee Badman*| Network Architect
Adjunct Instructor | CWNE #200
Information Technology Services
206 Machinery Hall
120 Smith Drive
Syracuse, New York 13244
*t*315.443.3003 *f* 315.443.4325 *e* [email protected]
<mailto:[email protected]> *w* its.syr.edu
*SYRACUSE UNIVERSITY*
syr.edu
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jason Cook
*Sent:* Tuesday, March 14, 2017 8:53 PM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Thanks for all the feedback, AVC was certainly the issue. As we don’t use this
it will remain turned off, should a requirement come up we’ll be certain to go
into plenty of discussion with Cisco.
We reloaded the controller last night and watched the load issues creep in this
morning. The first few on the graph below show up as slow icmp(40-60ms) response
for the client but throughput testing seemed ok. The one firing up just after 10
was where icmp for the client got above 100ms and throughput testing was
impacted. At it’s worse last week the graph below shows 800ms, clients were
often 400ms+ and was seen as bad as 1800ms. With throughput often 1-5 and
sometimes 0.
After graph I’ve provided some CLI from the client for ping and iperf for anyone
interested
On the plus side of this we got to update our code with very little change
control arguments JOn the down side it was a few days early as 8.2 MR5 isn’t out
yet L
*(Client based ping monitoring)*
64 bytes from 129.127.42.126: icmp_seq=71 ttl=255 time=167.135 ms
64 bytes from 129.127.42.126: icmp_seq=72 ttl=255 time=174.802 ms
64 bytes from 129.127.42.126: icmp_seq=73 ttl=255 time=185.832 ms
64 bytes from 129.127.42.126: icmp_seq=74 ttl=255 time=157.684 ms
64 bytes from 129.127.42.126: icmp_seq=75 ttl=255 time=166.068 ms
64 bytes from 129.127.42.126: icmp_seq=76 ttl=255 time=8.386 ms (AVC Disabled)
64 bytes from 129.127.42.126: icmp_seq=77 ttl=255 time=10.046 ms
64 bytes from 129.127.42.126: icmp_seq=78 ttl=255 time=3.454 ms
64 bytes from 129.127.42.126: icmp_seq=79 ttl=255 time=2.289 ms
64 bytes from 129.127.42.126: icmp_seq=80 ttl=255 time=2.384 ms
64 bytes from 129.127.42.126: icmp_seq=81 ttl=255 time=2.288 ms
64 bytes from 129.127.42.126: icmp_seq=82 ttl=255 time=2.502 ms
*(Client based throughput testing)*
student-10-201-00-173:~ networks$ iperf -c 129.127.41.37 -i 1 *(Pre-load-Issue)
AVC ON*
------------------------------------------------------------
Client connecting to 129.127.41.37, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 129.127.42.60 port 55631 connected with 129.127.41.37 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 9.62 MBytes 80.7 Mbits/sec
[ 4] 1.0- 2.0 sec 12.1 MBytes 102 Mbits/sec
[ 4] 2.0- 3.0 sec 14.2 MBytes 120 Mbits/sec
[ 4] 3.0- 4.0 sec 11.8 MBytes 98.6 Mbits/sec
[ 4] 4.0- 5.0 sec 14.2 MBytes 120 Mbits/sec
[ 4] 5.0- 6.0 sec 12.5 MBytes 105 Mbits/sec
[ 4] 6.0- 7.0 sec 13.2 MBytes 111 Mbits/sec
[ 4] 7.0- 8.0 sec 11.5 MBytes 96.5 Mbits/sec
[ 4] 8.0- 9.0 sec 12.4 MBytes 104 Mbits/sec
[ 4] 9.0-10.0 sec 12.4 MBytes 104 Mbits/sec
[ 4] 0.0-10.0 sec 124 MBytes 104 Mbits/sec
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$ date
Wed 15 Mar 2017 10:04:46 ACDT
student-10-201-00-173:~ networks$ iperf -c 129.127.41.37 -i 1 *(During-Issue)
AVC ON*
------------------------------------------------------------
Client connecting to 129.127.41.37, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 129.127.42.60 port 55710 connected with 129.127.41.37 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 256 KBytes 2.10 Mbits/sec
[ 4] 1.0- 2.0 sec 4.12 MBytes 34.6 Mbits/sec
[ 4] 2.0- 3.0 sec 9.50 MBytes 79.7 Mbits/sec
[ 4] 3.0- 4.0 sec 10.2 MBytes 86.0 Mbits/sec
[ 4] 4.0- 5.0 sec 7.38 MBytes 61.9 Mbits/sec
[ 4] 5.0- 6.0 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 6.0- 7.0 sec 2.75 MBytes 23.1 Mbits/sec
[ 4] 7.0- 8.0 sec 7.88 MBytes 66.1 Mbits/sec
[ 4] 8.0- 9.0 sec 9.00 MBytes 75.5 Mbits/sec
[ 4] 9.0-10.0 sec 9.38 MBytes 78.6 Mbits/sec
[ 4] 0.0-10.0 sec 61.8 MBytes 51.8 Mbits/sec
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$ iperf -c 129.127.41.37 -i 1 *(During-Issue)
AVC ON*
------------------------------------------------------------
Client connecting to 129.127.41.37, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 129.127.42.60 port 55730 connected with 129.127.41.37 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 384 KBytes 3.15 Mbits/sec
[ 4] 1.0- 2.0 sec 3.75 MBytes 31.5 Mbits/sec
[ 4] 2.0- 3.0 sec 6.25 MBytes 52.4 Mbits/sec
[ 4] 3.0- 4.0 sec 6.38 MBytes 53.5 Mbits/sec
[ 4] 4.0- 5.0 sec 5.75 MBytes 48.2 Mbits/sec
[ 4] 5.0- 6.0 sec 5.62 MBytes 47.2 Mbits/sec
[ 4] 6.0- 7.0 sec 5.50 MBytes 46.1 Mbits/sec
[ 4] 7.0- 8.0 sec 6.25 MBytes 52.4 Mbits/sec
[ 4] 8.0- 9.0 sec 6.25 MBytes 52.4 Mbits/sec
[ 4] 9.0-10.0 sec 9.50 MBytes 79.7 Mbits/sec
[ 4] 0.0-10.0 sec 55.8 MBytes 46.7 Mbits/sec
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$
student-10-201-00-173:~ networks$ iperf -c 129.127.41.37 -i 1 *(Post-Issue) AVC
OFF*
------------------------------------------------------------
Client connecting to 129.127.41.37, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 129.127.42.60 port 55740 connected with 129.127.41.37 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 10.8 MBytes 90.2 Mbits/sec
[ 4] 1.0- 2.0 sec 11.5 MBytes 96.5 Mbits/sec
[ 4] 2.0- 3.0 sec 13.5 MBytes 113 Mbits/sec
[ 4] 3.0- 4.0 sec 13.8 MBytes 115 Mbits/sec
[ 4] 4.0- 5.0 sec 14.0 MBytes 117 Mbits/sec
[ 4] 5.0- 6.0 sec 13.4 MBytes 112 Mbits/sec
[ 4] 6.0- 7.0 sec 14.1 MBytes 118 Mbits/sec
[ 4] 7.0- 8.0 sec 12.8 MBytes 107 Mbits/sec
[ 4] 8.0- 9.0 sec 10.6 MBytes 89.1 Mbits/sec
[ 4] 9.0-10.0 sec 14.1 MBytes 118 Mbits/sec
[ 4] 0.0-10.0 sec 129 MBytes 108 Mbits/sec
student-10-201-00-173:~ networks$ date
Wed 15 Mar 2017 10:08:59 ACDT
--
Jason Cook
Technology Services
The University of Adelaide, AUSTRALIA 5005
Ph : +61 8 8313 4800
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jason Cook
*Sent:* Friday, 10 March 2017 10:23 AM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
We have a good mix of AP’s. that’s interesting to know. thanks
--
Jason Cook
Technology Services
The University of Adelaide, AUSTRALIA 5005
Ph : +61 8 8313 4800
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jeffrey D. Sessler
*Sent:* Friday, 10 March 2017 2:19 AM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
AVC – turn it off. If you’re using older model WAPs, AVC imparts a high cost on
the controllers. On newer WAPs like the 3800, AVC is hardware accelerated.
If you are going to run 8.2, MR5 is due any day now and you’ll want the new
release.
Jeff
*From: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>> on behalf of Jake Snyder
<[email protected] <mailto:[email protected]>>
*Reply-To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Date: *Wednesday, March 8, 2017 at 10:13 PM
*To: *"[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
*Subject: *Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Might try leaving it off and see if that improves things. Just sounds oddly
familiar. Make sure you disable it on all SSIDs to make sure you get a fair
test with it off.
Sent from my iPhone
On Mar 8, 2017, at 10:56 PM, Jason Cook <[email protected]
<mailto:[email protected]>> wrote:
We don’t use it, but yes looking at our SSID config under QOS AVC is enabled
on. Is this the only place to enable it? Per SSID?
This would seem a good thing to kill off, clearly I should have paid more
attention to that discussion last year looking at history.
Thanks Jake
I’m now cringing a bit if this is the fix. Oh well. Gotta learn one way or
another
--
Jason Cook
Technology Services
The University of Adelaide, AUSTRALIA 5005
Ph : +61 8 8313 4800
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:[email protected]] *On Behalf Of *Jake Snyder
*Sent:* Thursday, 9 March 2017 4:08 PM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* Re: [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
I hate to ask, but do you have AVC enabled?
Sent from my iPhone
On Mar 8, 2017, at 9:59 PM, Watters, John <[email protected]
<mailto:[email protected]>> wrote:
I'll check the load on our most loaded 8510 HA pair in the morning & get
back to you. It is about 2300-2500 APs with at least that many
concurrent clients. Running 8.0.140.0 though (we moved there from a 7.6
(126 ?) level and Cisco recommended that we move to 8.0.140 before going
on up to 8.3).
We just bought a new 8510HA pair for this same MPLS area to divide the
load. It is running, but has no load at all yet. Was thinking of
starting it on 8.3 code. So, I am very interested in your problem and
tghe solution. Please be sure to post it.
==========================
-jcw
--------------------------------------------------------------------------------
*From:*The EDUCAUSE Wireless Issues Constituent Group Listserv
<[email protected]
<mailto:[email protected]>> on behalf of Jason Cook
<[email protected] <mailto:[email protected]>>
*Sent:* Wednesday, March 8, 2017 10:28:09 PM
*To:* [email protected]
<mailto:[email protected]>
*Subject:* [WIRELESS-LAN] Cisco 8510 8.2 Load Issues
Hi All,
Just wondering if anyone has had an similar experiences to the fun we’ve
had the last week or so.
Towards the end of last year we moved to new 8510 HA pair on 8.2.121.11
(we had an issue in testing at the time so grabbed the latest ER release
that resolved a crash bug)
From 5x5508’s in N+1 on 8.0.121.0 code
We started before the end of term with a small number of locations but
didn’t fully load it up until the big break. Now the students are back
and needing there internet we have had some real load issues during the
day.
SO it’s 2x 8510’s in HA about 2100 AP’s peaking at about 14k concurrent
clients but the issue seems to creep in at about 10k. While ICMP isn’t
the greatest tool for performance it does line up here, the graph below
show around 10am we see increased delays in response to the vlan42
(client network) interface on the controller and we see this on its
management interface too. At this point our clients ICMP to its own
gateway starts to increase from 1-3ms to 400-600 and even upto 1800
when the big spike shows 800ms to the interface. Iperf testing will also
go from 100Mb down to 1-5 and even 0 at times. With users complaining of
slowness and it’s worse unable to login.
CPU/Memory resources, channel util etc all ok. It’s site wide impact to
users no matter if it’s HD rf design or what AP model (1142,
2702,3702,3502 etc) So seems in the controller itself. All testing done
on 5hz
Around midday we started migrating AP’s away to our old 5508’s, which
saw a significant drop just before 12:30 and things back to normal at
12:40 once 300AP’s were moved off. So for now users are happy,
apparently we’ve even had callers in saying how good it is today (must
have been bad the last week for that to happen). Controller response to
SNMP was so bad it was taking Prime 2 minutes per AP to re-configure
primary controller. Did it by hand, ssh/gui response was not it’s normal
self but no problem. The 5508’s have shown no signs of being unhappy
with about 150 AP’s each.
We are working with TAC who have been good and they are investigating(no
like cases found though), shedding the load has worked around the issue
but it needs fixing. We upgraded to 8.2.141.0 yesterday evening but
won’t be re-loading the 8510’s until next week so confirming it’s fixed
is a few days off. There’s a few short upto 30ms delayed ICMP responses
today but it’s hard to know if that’s related or just the nature of icmp
and network gear priority.
Interested to know if anyone has seen anything like this in their
environment.
And anyone if anyone out there is using 8510’s in HA what’s your load in
AP and concurrent users? I can imagine many places loading their devices
up more than us
Anyone know how to look at other hardware resources (not
CPU/memory/system buffers) Something like ASIC on switches if it exists.
Surely all this traffic isn’t cpu
Thanks
Jason
<image001.jpg>
--
Jason Cook
Technology Services
The University of Adelaide, AUSTRALIA 5005
Ph : +61 8 8313 4800
e-mail: [email protected]<mailto:[email protected]
<mailto:[email protected]%3cmailto:[email protected]>>
CRICOS Provider Number 00123M
-----------------------------------------------------------
This email message is intended only for the addressee(s) and contains
information which may be confidential and/or copyright. If you are not
the intended recipient please do not read, save, forward, disclose, or
copy the contents of this email. If this email has been sent to you in
error, please notify the sender by reply email and delete this email and
any copies or links to this email completely and immediately from your
system. No representation is made that this email is free of viruses.
Virus scanning is recommended and is the responsibility of the
recipient.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
********** Participation and subscription information for this EDUCAUSE
Constituent Group discussion list can be found at
http://www.educause.edu/discuss.
