RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Chris Hart]

What rates did you set for the ARP policing?

Thanks

Chris Hart


[cid:image001.png@01D7A94B.455C09A0]
Chris Hart
Network Operations Engineer Lead
Tel: 847-467-7747
Email: ch...@northwestern.edu<mailto:ch...@northwestern.edu>
2020 Ridge Ave, Evanston, IL





From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Viou, Robert
Sent: Saturday, September 11, 2021 9:37 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

After working with the Aruba TAC last night, these are the changes we made that 
appear to have corrected the issues we were seeing.
Disabling Airgroup temporarily stops the issue with Queuing of the Arp packets. 
But the changes that we added below allowed us to re-enable Airgroup with 
APGroup set in the Profile.
Still need to monitor to be sure it is fixed, but so far looks good.

Monitor/police non-gratuitous ARP attacks: ENABLED
Monitor/police non-gratuitous ARP attack action: DROP



Bob

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 9:12 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office

On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://urldefense.com/v3/__https:/nam02.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.facebook.com*2FISUITHelp*2F=04*7C01*7CRobert.Viou*40NDSU.EDU*7Cdd0c720e506c47cda65308d9752e34d8*7Cec37a091b9a647e598d0903d4a419203*7C1*7C0*7C637669663628347496*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000=TP7NNp8n1*2BVyS2hYfqa7cYLY0bjswlO0FqAqTKioBQk*3D=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrRjZULnc$>
 and 
Twitter<https://urldefense.com/v3/__https:/nam02.safelinks.protection.outlook.com/?url=https*3A*2F*2Ftwitter.com*2FISUITHelp=04*7C01*7CRobert.Viou*40NDSU.EDU*7Cdd0c720e506c47cda65308d9752e34d8*7Cec37a091b9a647e598d0903d4a419203*7C1*7C0*7C637669663628357488*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000=RMTjQdg9p3bfKvhQcn*2BylQWZg2I*2FI3MyRPn31Qnh5rs*3D=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrg4Vpb6Q$>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Some people who received this message don't often get email from 
robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is 
important<https://urldefense.com/v3/__http:/aka.ms/LearnAboutSenderIdentification__;!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrBMxB_YE$>
[This message came from an external source. If suspicious, report to 
ab...@ilstu.

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Davis, Jonathan Alan]

e fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)
I'm not too familiar with how Aruba handles arps, does it do proxy arp? I have 
seen Apple devices go to sleep before all broadcast/multicast traffic is sent 
by the AP, although that was 5 years ago. So I can believe that a behaviour 
change could cause increased ARPs if the devices aren't seeing them.

Sent from my Galaxy



 Original message 
From: "Turner, Ryan H" 
Date: 12/9/21 09:16 (GMT+08:00)
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We actually are allowing MORE ARPs. Apparently when policing kicks in, all 
connections are affecting. It can cause clients to freeze/not connect.  So we 
actually turned the knob in the opposite direction.  We were seeing counters to 
what amounts to large quantities of controllers pauses when the ARPs went over 
an arbitrarily set number.  Our wireless architect can reply with the details.
Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office


On Sep 11, 2021, at 12:32 PM, Enfield, Chuck  wrote:

HI Ryan,

When you say that you detuned ARP policing, do you mean that the ARP policing 
on the underpinning network is now more aggressive (aka, dropping more ARP?)  I 
ask because I’ve been wondering why we aren’t seeing this problem when other 
schools that made the same changes we did still are.  We upgraded our 
underpinning network over the summer, and we’re dropping way more ARP than we 
were on the old network.  Your post just made me realize that may be protecting 
our controllers.  We’ve been considering changes, but we switched to an 
EVPN/VxLAN architecture.  We’re not completely sure what the consequences of 
this ARP policing is, so we’ve been holding off any changes.  If you had to 
police more aggressively to solve your problem, then we won’t start 
experimenting with out policers.

Thanks,

Chuck

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 10:12 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office

On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0>
 and 
Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[James Andrewartha]

I'm not too familiar with how Aruba handles arps, does it do proxy arp? I have 
seen Apple devices go to sleep before all broadcast/multicast traffic is sent 
by the AP, although that was 5 years ago. So I can believe that a behaviour 
change could cause increased ARPs if the devices aren't seeing them.

Sent from my Galaxy



 Original message 
From: "Turner, Ryan H" 
Date: 12/9/21 09:16 (GMT+08:00)
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We actually are allowing MORE ARPs. Apparently when policing kicks in, all 
connections are affecting. It can cause clients to freeze/not connect.  So we 
actually turned the knob in the opposite direction.  We were seeing counters to 
what amounts to large quantities of controllers pauses when the ARPs went over 
an arbitrarily set number.  Our wireless architect can reply with the details.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office

On Sep 11, 2021, at 12:32 PM, Enfield, Chuck  wrote:


HI Ryan,

When you say that you detuned ARP policing, do you mean that the ARP policing 
on the underpinning network is now more aggressive (aka, dropping more ARP?)  I 
ask because I’ve been wondering why we aren’t seeing this problem when other 
schools that made the same changes we did still are.  We upgraded our 
underpinning network over the summer, and we’re dropping way more ARP than we 
were on the old network.  Your post just made me realize that may be protecting 
our controllers.  We’ve been considering changes, but we switched to an 
EVPN/VxLAN architecture.  We’re not completely sure what the consequences of 
this ARP policing is, so we’ve been holding off any changes.  If you had to 
police more aggressively to solve your problem, then we won’t start 
experimenting with out policers.

Thanks,

Chuck

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 10:12 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office


On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0>
 and 
Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] An

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Turner, Ryan H]

We actually are allowing MORE ARPs. Apparently when policing kicks in, all 
connections are affecting. It can cause clients to freeze/not connect.  So we 
actually turned the knob in the opposite direction.  We were seeing counters to 
what amounts to large quantities of controllers pauses when the ARPs went over 
an arbitrarily set number.  Our wireless architect can reply with the details.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office

On Sep 11, 2021, at 12:32 PM, Enfield, Chuck  wrote:


HI Ryan,

When you say that you detuned ARP policing, do you mean that the ARP policing 
on the underpinning network is now more aggressive (aka, dropping more ARP?)  I 
ask because I’ve been wondering why we aren’t seeing this problem when other 
schools that made the same changes we did still are.  We upgraded our 
underpinning network over the summer, and we’re dropping way more ARP than we 
were on the old network.  Your post just made me realize that may be protecting 
our controllers.  We’ve been considering changes, but we switched to an 
EVPN/VxLAN architecture.  We’re not completely sure what the consequences of 
this ARP policing is, so we’ve been holding off any changes.  If you had to 
police more aggressively to solve your problem, then we won’t start 
experimenting with out policers.

Thanks,

Chuck

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 10:12 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office


On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0>
 and 
Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Some people who received this message don't often get email from 
robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is 
important<http://aka.ms/LearnAboutSenderIdentification>
[This message came from an external source. If suspicious, report to 
ab...@ilstu.edu<mailto:ab...@ilstu.edu>]
In regards to:
> Aruba believes this is the cause of the new iOS operating system.  Our 
> environment is extremely heavy iOS.  We are talking to them now and will 
> assess the change.

Has Aruba menti

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Viou, Robert]

After working with the Aruba TAC last night, these are the changes we made that 
appear to have corrected the issues we were seeing.
Disabling Airgroup temporarily stops the issue with Queuing of the Arp packets. 
But the changes that we added below allowed us to re-enable Airgroup with 
APGroup set in the Profile.
Still need to monitor to be sure it is fixed, but so far looks good.

Monitor/police non-gratuitous ARP attacks: ENABLED
Monitor/police non-gratuitous ARP attack action: DROP



Bob

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 9:12 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office


On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7CRobert.Viou%40NDSU.EDU%7Cdd0c720e506c47cda65308d9752e34d8%7Cec37a091b9a647e598d0903d4a419203%7C1%7C0%7C637669663628347496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=TP7NNp8n1%2BVyS2hYfqa7cYLY0bjswlO0FqAqTKioBQk%3D=0>
 and 
Twitter<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7CRobert.Viou%40NDSU.EDU%7Cdd0c720e506c47cda65308d9752e34d8%7Cec37a091b9a647e598d0903d4a419203%7C1%7C0%7C637669663628357488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=RMTjQdg9p3bfKvhQcn%2BylQWZg2I%2FI3MyRPn31Qnh5rs%3D=0>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Some people who received this message don't often get email from 
robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is 
important<http://aka.ms/LearnAboutSenderIdentification>
[This message came from an external source. If suspicious, report to 
ab...@ilstu.edu<mailto:ab...@ilstu.edu>]
In regards to:
> Aruba believes this is the cause of the new iOS operating system.  Our 
> environment is extremely heavy iOS.  We are talking to them now and will 
> assess the change.

Has Aruba mentioned to anyone or has anyone heard any more on this being an IOS 
issue?


Bob

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Norton, Thomas (Network Operations)
Sent: Wednesday, September 1, 2021 11:31 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Thanks for the update Ryan


T.J. Norton
Wireless Network Architect
Network Operations

(434) 592-6552

[http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg]

Liberty University  |  Training Champions for Christ since



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-L

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

HI Ryan,

When you say that you detuned ARP policing, do you mean that the ARP policing 
on the underpinning network is now more aggressive (aka, dropping more ARP?)  I 
ask because I’ve been wondering why we aren’t seeing this problem when other 
schools that made the same changes we did still are.  We upgraded our 
underpinning network over the summer, and we’re dropping way more ARP than we 
were on the old network.  Your post just made me realize that may be protecting 
our controllers.  We’ve been considering changes, but we switched to an 
EVPN/VxLAN architecture.  We’re not completely sure what the consequences of 
this ARP policing is, so we’ve been holding off any changes.  If you had to 
police more aggressively to solve your problem, then we won’t start 
experimenting with out policers.

Thanks,

Chuck

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Saturday, September 11, 2021 10:12 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We had to make major changes to bring stability to Khrushchev environment.  I 
think we have at this point.

We had to significantly detune the ARP policing policies.

We had to block virtually every SNMP poller.

We had to reboot our controllers.

We had to put in place an ACL to block communication from the Mobility masters.

A ridiculous amount of work to basically get us where we were 2 years ago and 
we probably have 15% lower connections compared to then.  I am hoping that the 
upcoming firmware fix will allow us to at least reverse the ACL and SNMP 
pollers. At this point we are pretty blind into information on individual 
connections.

Ryan Turner
Head of Networking, ITS
The University of North Carolina at Chapel Hill
+1 919 274 7926 Mobile
+1 919 445 0113 Office


On Sep 10, 2021, at 4:25 PM, Johnson, Christopher 
mailto:cbjo...@ilstu.edu>> wrote:

I haven’t heard anything as of yet. Although interestingly while doing a 
packet-capture to monitor arp/dhcp rates – noticed one client sending 
DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating 
constantly – and from the received signal strength of the client – there didn’t 
appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. 
So I’m wondering if that’s not an isolated behavior.

Christopher Johnson
Wireless Network Engineer
Office of Technology Solutions | Illinois State University
(309) 438-8444

Stay connected with ISU IT news and tips with @ISU IT Help on 
Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0>
 and 
Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Viou, Robert
Sent: Friday, September 10, 2021 10:28 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Some people who received this message don't often get email from 
robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is 
important<http://aka.ms/LearnAboutSenderIdentification>
[This message came from an external source. If suspicious, report to 
ab...@ilstu.edu<mailto:ab...@ilstu.edu>]
In regards to:
> Aruba believes this is the cause of the new iOS operating system.  Our 
> environment is extremely heavy iOS.  We are talking to them now and will 
> assess the change.

Has Aruba mentioned to anyone or has anyone heard any more on this being an IOS 
issue?


Bob

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Norton, Thomas (Network Operations)
Sent: Wednesday, September 1, 2021 11:31 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Thanks for the update Ryan


T.J. Norton

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Kenny, Eric]

Hi Chad,

We have an 8 node cluster with roughly 3,500 to 4,000 users on each node.  In 
this case “users” really means IP addresses the way they get counted.  So in a 
dual IPv4/IPv6 stack client environment, a single user may show up as 3 “users” 
with the IPv4 address, IPv6 link local, and globally routable IPv6 address.

Eric Kenny

Network Architect | Technology Partner Services
Harvard University Information Technology

On Sep 3, 2021, at 12:17 PM, Street, Chad A 
mailto:cstr...@emory.edu>> wrote:

For anyone who has applied the suggested fixes and are still having problems -- 
what are your client loads per controller?

I am also very curious on any feedback for anyone who has applied the fixes and 
has also successfully enabled Airwave ( not central ) without issues 
reoccurring.

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Patrick McEvilly 
mailto:patrick_mcevi...@harvard.edu>>
Sent: Friday, September 3, 2021 12:03 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

A quick update on where we are at.  We had many of the same issues being 
reported and did the same workarounds.  However, we ran into a few further 
issues that others might want to be aware of.

We had one AP group that did not have our AP crash dump server configured and 
was using the "default" which is the controller itself.  ~300 APs in that AP 
group had core dumped and were trying to write to the controller.  There is 
some mechanism that checks for available space on the controller before 
dumping.  This was jamming up the same queue that is used for other critical 
processes such as STM.  This caused a huge problem for us.  We changed this one 
AP group to point to our dump server and saw immediate relief.  All APs came 
back online and clients reconnected.

Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, 
not 17,000) on our wireless network.  We had a cluster event that triggered the 
controllers to send an invalid IPv4 address to all APs which in turn caused 
every AP to go into a core dump loop (that was not fun).  The only recovery 
option available was to reboot all controllers.  It is believed this is related 
to 8.7 now supporting IPv6 in the clustering configuration.  Our controllers 
are dual stack but our APs are not.  It is possible if the APs were dual 
stacked they would have used IPv6 and continued to operate over IPv6.  Per 
Aruba removing IPv6 from the controllers is also not an option, we could still 
experience this problem.  This one was a real burn and we continue to be 
susceptible to this condition.

We are anticipating an emergency code release from Aruba in the next few days.


Patrick






On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Jerry Bucklaew" 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
on behalf of j...@buffalo.edu<mailto:j...@buffalo.edu>> wrote:

Scott,

  I have heard it worked in several places, but we are like you.  Put in 
all the recommended changes last night and it is still happening.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
    Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been 
discussing?  We had Aruba on the phone very early this morning - put the 
recommended ACL in place and did a a rolling restart of all 3 of our 
controllers.  We only had to wait to our first class change to see a huge drop 
of clients and another at the next class change.  Seems clear that the change 
and the reboot made no difference.  back on the phone with Aruba now to see 
what else they come up with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu<mailto:sco...@wcu.edu>


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent th

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Jerry Bucklaew]

We have two cluster, one with issues and one without.  On the one with issues 
we are seeing roughly 4500 clients per controller.  On the one without issues 
we are seeing around 2000 clients per controller.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Friday, September 3, 2021 12:17 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

For anyone who has applied the suggested fixes and are still having problems -- 
what are your client loads per controller?

I am also very curious on any feedback for anyone who has applied the fixes and 
has also successfully enabled Airwave ( not central ) without issues 
reoccurring.

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Patrick McEvilly 
mailto:patrick_mcevi...@harvard.edu>>
Sent: Friday, September 3, 2021 12:03 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

A quick update on where we are at.  We had many of the same issues being 
reported and did the same workarounds.  However, we ran into a few further 
issues that others might want to be aware of.

We had one AP group that did not have our AP crash dump server configured and 
was using the "default" which is the controller itself.  ~300 APs in that AP 
group had core dumped and were trying to write to the controller.  There is 
some mechanism that checks for available space on the controller before 
dumping.  This was jamming up the same queue that is used for other critical 
processes such as STM.  This caused a huge problem for us.  We changed this one 
AP group to point to our dump server and saw immediate relief.  All APs came 
back online and clients reconnected.

Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, 
not 17,000) on our wireless network.  We had a cluster event that triggered the 
controllers to send an invalid IPv4 address to all APs which in turn caused 
every AP to go into a core dump loop (that was not fun).  The only recovery 
option available was to reboot all controllers.  It is believed this is related 
to 8.7 now supporting IPv6 in the clustering configuration.  Our controllers 
are dual stack but our APs are not.  It is possible if the APs were dual 
stacked they would have used IPv6 and continued to operate over IPv6.  Per 
Aruba removing IPv6 from the controllers is also not an option, we could still 
experience this problem.  This one was a real burn and we continue to be 
susceptible to this condition.

We are anticipating an emergency code release from Aruba in the next few days.


Patrick






On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Jerry Bucklaew" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU%20on%20behalf%20of%20...@buffalo.edu>>
 wrote:

Scott,

  I have heard it worked in several places, but we are like you.  Put in 
all the recommended changes last night and it is still happening.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
    Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been 
discussing?  We had Aruba on the phone very early this morning - put the 
recommended ACL in place and did a a rolling restart of all 3 of our 
controllers.  We only had to wait to our first class change to see a huge drop 
of clients and another at the next class change.  Seems clear that the change 
and the reboot made no difference.  back on the phone with Aruba now to see 
what else they come up with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu<mailto:sco...@wcu.edu>


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who s

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Street, Chad A]

For anyone who has applied the suggested fixes and are still having problems -- 
what are your client loads per controller?

I am also very curious on any feedback for anyone who has applied the fixes and 
has also successfully enabled Airwave ( not central ) without issues 
reoccurring.

Chad
chad.str...@emory.edu

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Patrick McEvilly 

Sent: Friday, September 3, 2021 12:03 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

A quick update on where we are at.  We had many of the same issues being 
reported and did the same workarounds.  However, we ran into a few further 
issues that others might want to be aware of.

We had one AP group that did not have our AP crash dump server configured and 
was using the "default" which is the controller itself.  ~300 APs in that AP 
group had core dumped and were trying to write to the controller.  There is 
some mechanism that checks for available space on the controller before 
dumping.  This was jamming up the same queue that is used for other critical 
processes such as STM.  This caused a huge problem for us.  We changed this one 
AP group to point to our dump server and saw immediate relief.  All APs came 
back online and clients reconnected.

Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, 
not 17,000) on our wireless network.  We had a cluster event that triggered the 
controllers to send an invalid IPv4 address to all APs which in turn caused 
every AP to go into a core dump loop (that was not fun).  The only recovery 
option available was to reboot all controllers.  It is believed this is related 
to 8.7 now supporting IPv6 in the clustering configuration.  Our controllers 
are dual stack but our APs are not.  It is possible if the APs were dual 
stacked they would have used IPv6 and continued to operate over IPv6.  Per 
Aruba removing IPv6 from the controllers is also not an option, we could still 
experience this problem.  This one was a real burn and we continue to be 
susceptible to this condition.

We are anticipating an emergency code release from Aruba in the next few days.


Patrick






On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Jerry Bucklaew"  wrote:

Scott,

  I have heard it worked in several places, but we are like you.  Put in 
all the recommended changes last night and it is still happening.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been 
discussing?  We had Aruba on the phone very early this morning - put the 
recommended ACL in place and did a a rolling restart of all 3 of our 
controllers.  We only had to wait to our first class change to see a huge drop 
of clients and another at the next class change.  Seems clear that the change 
and the reboot made no difference.  back on the phone with Aruba now to see 
what else they come up with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccstree2%40EMORY.EDU%7C7010b9354b0144becdd908d96ef458f7%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C637662818017617110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=Py3VDUVjpo1WIQ0y8v3PmIk%2BhoQi4TbaMfP0bDNopCw%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccstree2%40EMORY.EDU%7C7010b9354b01

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Patrick McEvilly]

A quick update on where we are at.  We had many of the same issues being 
reported and did the same workarounds.  However, we ran into a few further 
issues that others might want to be aware of.

We had one AP group that did not have our AP crash dump server configured and 
was using the "default" which is the controller itself.  ~300 APs in that AP 
group had core dumped and were trying to write to the controller.  There is 
some mechanism that checks for available space on the controller before 
dumping.  This was jamming up the same queue that is used for other critical 
processes such as STM.  This caused a huge problem for us.  We changed this one 
AP group to point to our dump server and saw immediate relief.  All APs came 
back online and clients reconnected.  

Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, 
not 17,000) on our wireless network.  We had a cluster event that triggered the 
controllers to send an invalid IPv4 address to all APs which in turn caused 
every AP to go into a core dump loop (that was not fun).  The only recovery 
option available was to reboot all controllers.  It is believed this is related 
to 8.7 now supporting IPv6 in the clustering configuration.  Our controllers 
are dual stack but our APs are not.  It is possible if the APs were dual 
stacked they would have used IPv6 and continued to operate over IPv6.  Per 
Aruba removing IPv6 from the controllers is also not an option, we could still 
experience this problem.  This one was a real burn and we continue to be 
susceptible to this condition.

We are anticipating an emergency code release from Aruba in the next few days.


Patrick






On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Jerry Bucklaew"  wrote:

Scott, 

  I have heard it worked in several places, but we are like you.  Put in 
all the recommended changes last night and it is still happening.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been 
discussing?  We had Aruba on the phone very early this morning - put the 
recommended ACL in place and did a a rolling restart of all 3 of our 
controllers.  We only had to wait to our first class change to see a huge drop 
of clients and another at the next class change.  Seems clear that the change 
and the reboot made no difference.  back on the phone with Aruba now to see 
what else they come up with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cjpb%40buffalo.edu%7Cf58d662303514aa7e05808d96eea39ec%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637662774536265169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=QcSqYr93it4Sn%2Fc1GaKHK4AGNRz77g7qR8SZ4EbPpKA%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Jerry Bucklaew]

Scott, 

  I have heard it worked in several places, but we are like you.  Put in all 
the recommended changes last night and it is still happening.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been discussing?  
We had Aruba on the phone very early this morning - put the recommended ACL in 
place and did a a rolling restart of all 3 of our controllers.  We only had to 
wait to our first class change to see a huge drop of clients and another at the 
next class change.  Seems clear that the change and the reboot made no 
difference.  back on the phone with Aruba now to see what else they come up 
with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cjpb%40buffalo.edu%7Cf58d662303514aa7e05808d96eea39ec%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637662774536265169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=QcSqYr93it4Sn%2Fc1GaKHK4AGNRz77g7qR8SZ4EbPpKA%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Ortman, Wayne]

I don't want to jinx anything but we've been able to keep the issues away for 
about 10 days now.  In addition to the ACL changes we've also enabled the 
broadcast/multicast optimization, lowered our client rebalancing thresholds, 
and turn off SNMP for the most part.

We're leaning to the client rebalancing being one of the bigger portions of 
that. Before changing that we had some controllers with 8k clients and some 
with 10.

Wayne Ortman
Director, Network Services
Office of Information Technology  (OIT)
1762 Clifton Road |E154
Office (404) 727-8014 | Cell (470) 312-5754 |wayne.ort...@emory.edu
 
 


-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Scott Swartzentruber
Sent: Friday, September 3, 2021 10:41 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Has anyone had success in resolving the client drops we have been discussing?  
We had Aruba on the phone very early this morning - put the recommended ACL in 
place and did a a rolling restart of all 3 of our controllers.  We only had to 
wait to our first class change to see a huge drop of clients and another at the 
next class change.  Seems clear that the change and the reboot made no 
difference.  back on the phone with Aruba now to see what else they come up 
with...


I don't like this ride anymore - can I get off now?



Western Carolina University
Scott Swartzentruber
Director, Networking and Communications
Forsyth Bldg, Room B-16
828.227.3212
sco...@wcu.edu


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cwayne.ortman%40EMORY.EDU%7C770659141b174cfc57d008d96eea3937%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C637662774592961015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=0yX3iNp6ytwHdhV8JRa8nHQ9SxQj464pArCRAPuoblo%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Colin Randall]

Sorry I’m late to the party.  I got ahold of a great tech at Aruba last night.  
We finished up with his suggested work-arounds 2-3 hours before the advisory 
came out.
Implemented the client rebalancing threshold change, also at the seemingly 
popular 15%
Turned on ALL the broadcast/multicast optimization switches, at both the VLAN 
and virtual-ap level.
Tweaked the mentioned ACLs (but I doubt that did much.  An ACL would just 
prevent sending the traffic, not wasting cycles generating the traffic.)
Tech mentioned SNMP polling could be an issue, but since we’re preferring AMON 
we left that one alone.
Lastly, restarted STM process on each controller.  Definitely service impacting 
but staggering the restart across multiple controllers seemed to help.
I hadn’t seen a few of those in the thread.

Regards,

Colin

 

 

Colin Randall

Manager of Data Networking

Information and Technology Solutions (ITS)

1600 Jackson Street, Suite 360

303-384-2208 | crand...@mines.edu

 

 

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of "Enfield, Chuck" 

Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 

Date: Thursday, September 2, 2021 at 11:06 AM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

 

CAUTION: This email originated from outside of the Colorado School of Mines 
organization. Do not click on links or open attachments unless you recognize 
the sender and know the content is safe.

 

I’d like to suggest sending them home, but if we learned anything last year 
it’s that home wi-fi isn’t so great either.  How many times have you heard, “It 
works when I’m at home?”  Well now we know, not always.

 

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Floyd, Brad
Sent: Thursday, September 2, 2021 1:00 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

 

JD,

If we wrap each of them along with their devices in an aluminum foil bubble, 
each user would have their own collision domain. The MIMO reflections would be 
awesome, we wouldn’t need more than a single channel architecture, and any 
channel contention would be self-imposed. Here’s hoping we get to catch up 
again at the next post-COVID WLPC.

Thanks,

Brad

 

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Davis, Jonathan Alan
Sent: Thursday, September 2, 2021 11:42 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

 

[EXTERNAL SENDER]

“That's been my experience for years.  The network works great when there are 
no students around.  My working theory is that students emit RF interference, 
but research ethics won’t let me run the tests, so we'll never know for sure.”

 

It’s worse than that! They are walking bags of water which absorb the good RF, 
and their devices transmit the bad RF! It’s a conspiracy I tell ya!

 

We’re going to work with TAC on capturing traffic during a class that is known 
to have issues. After that, we plan to change the rebalancing threshold as 
well. 

 

Thanks everyone for the feedback!

 

JD

 

 

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Enfield, Chuck 

Date: Thursday, September 2, 2021 at 12:15 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

I will also add that our problems did not increase linearly with client count 
on a controller.  Below 5K there was no user impact.  Around 5K problems 
started and the severity increased quickly.  I doubt there’s anything magic 
about 5K, and the threshold will be different on every network based on a 
variety of implementation details, but I’d expect that pattern to be common.

 

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Enfield, Chuck
Sent: Thursday, September 2, 2021 11:21 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

 

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK. 

From: "Street, Chad A" 
Sent: Thursday, September 2, 2021 11:03 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection 

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Lee H Badman]

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”… 

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck  wrote:
> 
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
> 
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
> 
> Speaking from experience, I would be very concerned.  We had no issues until 
> students returned and we went downhill from there.
> 
> 
> On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv 
> on behalf of Rob Harris"  robert.har...@culinary.edu> wrote:
> 
>Has anyone seen any details regarding what they consider "Large" 
> environments? We upgraded during the break, but both before and after 
> versions are affected. We didn't notice this happening before, should we be 
> concerned now?
> 
>The "dropped" is 0 and the stm cpu usage is in single digits, but client 
> count is really low (they come back this weekend as well), could we be in the 
> clear?
> 
>(asked the SE team and opened a tac call, same questions to them)
> 
>thx
> 
>-Original Message-
>From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Jason Healy
>    Sent: Thursday, September 2, 2021 8:45 AM
>    To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
> seeing any issues in the fall with large classrooms and delayed connection 
> times (Aruba 8.5.0.13)
> 
>CAUTION: This email originated from outside The Culinary Institute of 
> America. Do not click links or open attachments unless you recognize the 
> sender and know the content is safe.
> 
>FWIW, Aruba just posted an advisory regarding this issue:
> 
>Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client 
> Connectivity Failures in Large Client Environments"
> 
>Good luck to those of you hit by this. My students start coming back this 
> weekend so I'll be watching this closely!
> 
>Jason
>**
>Replies to EDUCAUSE Community Group emails are sent to the entire 
> community list. If you want to reply only to the person who sent the message, 
> copy and paste their email address and forward the email reply. Additional 
> participation and subscription information can be found at 
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0
> 
>**
>Replies to EDUCAUSE Community Group emails are sent to the entire 
> community list. If you want to reply only to the person who sent the message, 
> copy and paste their email address and forward the email reply. Additional 
> participation and subscription information can be found at 
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0
> 
> **
> Replies to EDUCAUSE Community Group emails are sent to the entire community 
> list. If you want to reply only to the person who sent the message, copy and 
> paste their email address and forward the email reply. Additional 
> participation and subscription information can be found at 
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

I’d like to suggest sending them home, but if we learned anything last year 
it’s that home wi-fi isn’t so great either.  How many times have you heard, “It 
works when I’m at home?”  Well now we know, not always.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Floyd, Brad
Sent: Thursday, September 2, 2021 1:00 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

JD,
If we wrap each of them along with their devices in an aluminum foil bubble, 
each user would have their own collision domain. The MIMO reflections would be 
awesome, we wouldn’t need more than a single channel architecture, and any 
channel contention would be self-imposed. Here’s hoping we get to catch up 
again at the next post-COVID WLPC.
Thanks,
Brad

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Davis, Jonathan Alan
Sent: Thursday, September 2, 2021 11:42 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


[EXTERNAL SENDER]
“That's been my experience for years.  The network works great when there are 
no students around.  My working theory is that students emit RF interference, 
but research ethics won’t let me run the tests, so we'll never know for sure.”

It’s worse than that! They are walking bags of water which absorb the good RF, 
and their devices transmit the bad RF! It’s a conspiracy I tell ya!

We’re going to work with TAC on capturing traffic during a class that is known 
to have issues. After that, we plan to change the rebalancing threshold as well.

Thanks everyone for the feedback!

JD


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Enfield, Chuck mailto:cae...@psu.edu>>
Date: Thursday, September 2, 2021 at 12:15 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)
I will also add that our problems did not increase linearly with client count 
on a controller.  Below 5K there was no user impact.  Around 5K problems 
started and the severity increased quickly.  I doubt there’s anything magic 
about 5K, and the threshold will be different on every network based on a 
variety of implementation details, but I’d expect that pattern to be common.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Enfield, Chuck
Sent: Thursday, September 2, 2021 11:21 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK.

From: "Street, Chad A" mailto:cstr...@emory.edu>>
Sent: Thursday, September 2, 2021 11:03 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Rob Harris 
mailto:robert.har...@culinary.edu>>
Sent: Thursday, September 2, 2021 10:59 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Floyd, Brad]

JD,
If we wrap each of them along with their devices in an aluminum foil bubble, 
each user would have their own collision domain. The MIMO reflections would be 
awesome, we wouldn’t need more than a single channel architecture, and any 
channel contention would be self-imposed. Here’s hoping we get to catch up 
again at the next post-COVID WLPC.
Thanks,
Brad

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Davis, Jonathan Alan
Sent: Thursday, September 2, 2021 11:42 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


[EXTERNAL SENDER]
“That's been my experience for years.  The network works great when there are 
no students around.  My working theory is that students emit RF interference, 
but research ethics won’t let me run the tests, so we'll never know for sure.”

It’s worse than that! They are walking bags of water which absorb the good RF, 
and their devices transmit the bad RF! It’s a conspiracy I tell ya!

We’re going to work with TAC on capturing traffic during a class that is known 
to have issues. After that, we plan to change the rebalancing threshold as well.

Thanks everyone for the feedback!

JD


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Enfield, Chuck mailto:cae...@psu.edu>>
Date: Thursday, September 2, 2021 at 12:15 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)
I will also add that our problems did not increase linearly with client count 
on a controller.  Below 5K there was no user impact.  Around 5K problems 
started and the severity increased quickly.  I doubt there’s anything magic 
about 5K, and the threshold will be different on every network based on a 
variety of implementation details, but I’d expect that pattern to be common.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Enfield, Chuck
Sent: Thursday, September 2, 2021 11:21 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK.

From: "Street, Chad A" mailto:cstr...@emory.edu>>
Sent: Thursday, September 2, 2021 11:03 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Rob Harris 
mailto:robert.har...@culinary.edu>>
Sent: Thursday, September 2, 2021 10:59 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in t

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Davis, Jonathan Alan]

“That's been my experience for years.  The network works great when there are 
no students around.  My working theory is that students emit RF interference, 
but research ethics won’t let me run the tests, so we'll never know for sure.”

It’s worse than that! They are walking bags of water which absorb the good RF, 
and their devices transmit the bad RF! It’s a conspiracy I tell ya!

We’re going to work with TAC on capturing traffic during a class that is known 
to have issues. After that, we plan to change the rebalancing threshold as well.

Thanks everyone for the feedback!

JD


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Enfield, Chuck 

Date: Thursday, September 2, 2021 at 12:15 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)
I will also add that our problems did not increase linearly with client count 
on a controller.  Below 5K there was no user impact.  Around 5K problems 
started and the severity increased quickly.  I doubt there’s anything magic 
about 5K, and the threshold will be different on every network based on a 
variety of implementation details, but I’d expect that pattern to be common.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Enfield, Chuck
Sent: Thursday, September 2, 2021 11:21 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK.

From: "Street, Chad A" mailto:cstr...@emory.edu>>
Sent: Thursday, September 2, 2021 11:03 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Rob Harris 
mailto:robert.har...@culinary.edu>>
Sent: Thursday, September 2, 2021 10:59 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image001.png@01D79FF7.DF429460]



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

I will also add that our problems did not increase linearly with client count 
on a controller.  Below 5K there was no user impact.  Around 5K problems 
started and the severity increased quickly.  I doubt there’s anything magic 
about 5K, and the threshold will be different on every network based on a 
variety of implementation details, but I’d expect that pattern to be common.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Enfield, Chuck
Sent: Thursday, September 2, 2021 11:21 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK.

From: "Street, Chad A" mailto:cstr...@emory.edu>>
Sent: Thursday, September 2, 2021 11:03 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Rob Harris 
mailto:robert.har...@culinary.edu>>
Sent: Thursday, September 2, 2021 10:59 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image001.png@01D79FF4.0F7F8150]



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



Lee, don’t you bring your bad Cisco-juju to this conversation! :-)



Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.



Hello!



Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.



This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Barrantes, Rita]

We are on 8.7.1.4 with 6,300 WAPs and 6 controllers. We had to upgrade to 8.7 
because of the newer model of WAPs


___
Rita Barrantes, PhD, PMP
Director, Technology Services and Support
Networks and Telecomm | IT Assessments
832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu>
[cid:image001.png@01D79FE7.35A3D4B0]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Joe Walker
Sent: Thursday, September 2, 2021 10:16 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We currently aren't experiencing this issue but these symptoms (stm crashes, 
controller crashes) were identical to the issues we saw in early 2019 (8.5.0.1) 
An aggressive rebalance of users/ap's (and eventually an upgrade to 8.5.0.3) 
fixed this for us.  I mention this because we are currently sitting on 8.5.0.13 
and don't seem to be having the issues reported by others in the same code so I 
wonder if there is credence to the load balance thresholds playing a part.

Campus cluster 2x7240's 1X7240XM
22K connected devices

Thanks,
Joe

Joe Walker
Network and Telecommunication Services
Kansas State University
(785)532-4997
f...@ksu.edu<mailto:f...@ksu.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Smith, Nayef 
mailto:nayef.z.sm...@emory.edu>>
Sent: Thursday, September 2, 2021 9:19 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


This email originated from outside of K-State.

"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."


To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.


We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.


Nayef Z. Smith | Network Services | Voice: 404-727-6019


[cid:image002.png@01D79FE7.35A3D4B0]


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.


Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But y

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Barrantes, Rita]

Nayef – that’s what we did. Down to 15%


___
Rita Barrantes, PhD, PMP
Director, Technology Services and Support
Networks and Telecomm | IT Assessments
832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu>
[cid:image002.png@01D79FE6.EC8BFA50]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:13 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We were seeing one of our cluster members taking on ~7000+ active clients while 
two others were in the low double digits.  Our previous threshold was at 50%, 
which seems to have previously been the default.  Once we adjusted our 
thresholds down to 20%, we saw improvements.  We then went down to 15%.




Nayef Z. Smith | Network Services | Voice: 404-727-6019


[cid:image003.png@01D79FE6.EC8BFA50]


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Rob Harris 
mailto:robert.har...@culinary.edu>>
Sent: Thursday, September 2, 2021 10:59 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image003.png@01D79FE6.EC8BFA50]



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



Lee, don’t you bring your bad Cisco-juju to this conversation! :-)



Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.



Hello!



Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.



This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.



However…

We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.

I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.



I’ll update as we progress through this.



JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<ma

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Charles Jouglard]

Any chance you could be experiencing the issues outlined in today's
advisory?

https://mail.google.com/mail/u/0?ui=2=b16ec71188=0.1=msg-f:1709803220174325554=17ba70f8464f4732=att=inline

On Thu, Sep 2, 2021 at 10:20 AM Enfield, Chuck  wrote:

> Between 5k and 6k clients on a 7240xm is where we started seeing problems.
> Lighter loaded controllers were OK.
> --
> *From:* "Street, Chad A" 
> *Sent:* Thursday, September 2, 2021 11:03 AM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
> seeing any issues in the fall with large classrooms and delayed connection
> times (Aruba 8.5.0.13)
>
> We are a balanced cluster, notes about load below:
>
>
> "I’m also noticing that there are much fewer clients on this controller,
> and that ratio doesn’t seem to be improving."
>
> To this point, the action we took that seemed to help the most was
> adjusting our active client load balancing threshold.  We dropped it
> significantly to force clients to balance across controllers.  Once we got
> below ~5000 active clients per controller, we stopped seeing the mass
> client connection issues.
>
> We still have a controller that hasn't taken significant load, but now
> that we've been running without major issues for the past few days, we're
> reluctant to touch the setting again.
> --
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Rob Harris <
> robert.har...@culinary.edu>
> *Sent:* Thursday, September 2, 2021 10:59 AM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
> seeing any issues in the fall with large classrooms and delayed connection
> times (Aruba 8.5.0.13)
>
>
> For those of you who have experienced this, what was your user load and
> how were your clusters operating (balancing, active/standby) ?
>
>
>
> I wonder if there’s a threshold..
>
>
>
> Thx!
>
>
>
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Smith, Nayef
> *Sent:* Thursday, September 2, 2021 10:20 AM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
> seeing any issues in the fall with large classrooms and delayed connection
> times (Aruba 8.5.0.13)
>
>
>
>
>
>
>
> "I’m also noticing that there are much fewer clients on this controller,
> and that ratio doesn’t seem to be improving."
>
>
>
> To this point, the action we took that seemed to help the most was
> adjusting our active client load balancing threshold.  We dropped it
> significantly to force clients to balance across controllers.  Once we got
> below ~5000 active clients per controller, we stopped seeing the mass
> client connection issues.
>
>
>
> We still have a controller that hasn't taken significant load, but now
> that we've been running without major issues for the past few days, we're
> reluctant to touch the setting again.
>
>
>
>
>
> Nayef Z. Smith | *Network Services* | Voice: 404-727-6019
>
>
>
>
>
> ----------
>
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Davis, Jonathan Alan <
> jonath...@unc.edu>
> *Sent:* Thursday, September 2, 2021 9:27 AM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
> seeing any issues in the fall with large classrooms and delayed connection
> times (Aruba 8.5.0.13)
>
>
>
> Lee, don’t you bring your bad Cisco-juju to this conversation! :-)
>
>
>
> Now that Lee has been properly handled, this is probably a great
> opportunity to say ‘hello’ to the greater list.
>
>
>
> Hello!
>
>
>
> Last night, we (UNC) restarted the controller used to test the firewall
> policy. Despite Aruba’s advisory, we’ve been led to believe that restarting
> STM may not be enough, and restarting the whole controller may be required
> to resolve high STM CPU utilization.
>
>
>
> This morning we are keeping a close eye on that controller. While STM is
> surging well past 100%, it seems to be averaging much closer to 95%.
>
>
>
> However…
>
> We also only have about 7,000 users connected across the cluster. It will
> be interesting to see what happens as the day progresses and students wak

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

Between 5k and 6k clients on a 7240xm is where we started seeing problems. 
Lighter loaded controllers were OK.

From: "Street, Chad A" 
Sent: Thursday, September 2, 2021 11:03 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Rob Harris 

Sent: Thursday, September 2, 2021 10:59 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image001.png@01D79FE9.A97E6BF0]




From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



Lee, don’t you bring your bad Cisco-juju to this conversation! :-)



Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.



Hello!



Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.



This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.



However…

We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.

I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.



I’ll update as we progress through this.



JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large cla

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Joe Walker]

We currently aren't experiencing this issue but these symptoms (stm crashes, 
controller crashes) were identical to the issues we saw in early 2019 (8.5.0.1) 
An aggressive rebalance of users/ap's (and eventually an upgrade to 8.5.0.3) 
fixed this for us.  I mention this because we are currently sitting on 8.5.0.13 
and don't seem to be having the issues reported by others in the same code so I 
wonder if there is credence to the load balance thresholds playing a part.

Campus cluster 2x7240's 1X7240XM
22K connected devices

Thanks,
Joe

Joe Walker
Network and Telecommunication Services
Kansas State University
(785)532-4997
f...@ksu.edu<mailto:f...@ksu.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Smith, Nayef 

Sent: Thursday, September 2, 2021 9:19 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


This email originated from outside of K-State.

"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.


Nayef Z. Smith | Network Services | Voice: 404-727-6019

[cid:dd82bb8e-7a57-4a56-8047-d94ceebd7bec]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Davis, Jonathan Alan 

Sent: Thursday, September 2, 2021 9:27 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.

Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Lee H Badman 

Sent: Thursday, September 2, 2021 9:06:33 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck  wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
>
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
>
> Speaking from experience, I would be very concerned.  We had no issues until 
> stu

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Smith, Nayef]

We were seeing one of our cluster members taking on ~7000+ active clients while 
two others were in the low double digits.  Our previous threshold was at 50%, 
which seems to have previously been the default.  Once we adjusted our 
thresholds down to 20%, we saw improvements.  We then went down to 15%.




Nayef Z. Smith | Network Services | Voice: 404-727-6019

[cid:d4681667-5ec0-49ec-b547-ee2c893c1e10]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Rob Harris 

Sent: Thursday, September 2, 2021 10:59 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image001.png@01D79FE9.A97E6BF0]




From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



Lee, don’t you bring your bad Cisco-juju to this conversation! :-)



Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.



Hello!



Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.



This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.



However…

We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.

I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.



I’ll update as we progress through this.



JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck 
> mailto:cae...@psu.edu>> wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is 

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Barrantes, Rita]

We are experiencing the same issues. We rebooted this morning.


___
Rita Barrantes, PhD, PMP
Director, Technology Services and Support
Networks and Telecomm | IT Assessments
832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu>
[cid:image001.png@01D79FE2.62D01BB0]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 9:20 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.


Nayef Z. Smith | Network Services | Voice: 404-727-6019


[cid:image002.png@01D79FE2.62D01BB0]


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.


Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck 
> mailto:cae...@psu.edu>> wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
>
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: 
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
>
> Speaking from experience, I would be very 

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Street, Chad A]

We are a balanced cluster, notes about load below:


"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Rob Harris 

Sent: Thursday, September 2, 2021 10:59 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?



I wonder if there’s a threshold..



Thx!



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)







"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."



To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.



We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.





Nayef Z. Smith | Network Services | Voice: 404-727-6019



[cid:image001.png@01D79FE9.A97E6BF0]




From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



Lee, don’t you bring your bad Cisco-juju to this conversation! :-)



Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.



Hello!



Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.



This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.



However…

We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.

I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.



I’ll update as we progress through this.



JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)



But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Rob Harris]

For those of you who have experienced this, what was your user load and how 
were your clusters operating (balancing, active/standby) ?

I wonder if there’s a threshold..

Thx!

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Smith, Nayef
Sent: Thursday, September 2, 2021 10:20 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)




"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.


Nayef Z. Smith | Network Services | Voice: 404-727-6019


[cid:image001.png@01D79FE9.A97E6BF0]


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>>
Sent: Thursday, September 2, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.


Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Lee H Badman mailto:lhbad...@syr.edu>>
Sent: Thursday, September 2, 2021 9:06:33 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck 
> mailto:cae...@psu.edu>> wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
>
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: 
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
>
> Speaking from experience, I would be very concerned.  We had no issues until 
> students returned and we went downhill from there.
>
>
> 

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Smith, Nayef]

"I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving."

To this point, the action we took that seemed to help the most was adjusting 
our active client load balancing threshold.  We dropped it significantly to 
force clients to balance across controllers.  Once we got below ~5000 active 
clients per controller, we stopped seeing the mass client connection issues.

We still have a controller that hasn't taken significant load, but now that 
we've been running without major issues for the past few days, we're reluctant 
to touch the setting again.


Nayef Z. Smith | Network Services | Voice: 404-727-6019

[cid:dd82bb8e-7a57-4a56-8047-d94ceebd7bec]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Davis, Jonathan Alan 

Sent: Thursday, September 2, 2021 9:27 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.

Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Lee H Badman 

Sent: Thursday, September 2, 2021 9:06:33 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck  wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
>
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
>
> Speaking from experience, I would be very concerned.  We had no issues until 
> students returned and we went downhill from there.
>
>
> On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv 
> on behalf of Rob Harris"  robert.har...@culinary.edu> wrote:
>
>Has anyone seen any details regarding what they consider "Large" 
> environments? We upgraded during the break, but both before and after 
> versions are affected. We didn't notice this happening before, should we be 
> concerned now?
>
>The "dropped" is 0 and the stm cpu usage is in single digits, but client 
> count is really low (they come back this weekend as well), could we be in the 
> clear?
>
>(asked the SE team and opened a tac call, same questions to them)
>
>thx
>
>-Original Message-
>From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Jason Healy
>    Sent: Thursday, September 2, 2021 8:45 AM
>    To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
>

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Davis, Jonathan Alan]

Lee, don’t you bring your bad Cisco-juju to this conversation! :-)

Now that Lee has been properly handled, this is probably a great opportunity to 
say ‘hello’ to the greater list.

Hello!

Last night, we (UNC) restarted the controller used to test the firewall policy. 
Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not 
be enough, and restarting the whole controller may be required to resolve high 
STM CPU utilization.

This morning we are keeping a close eye on that controller. While STM is 
surging well past 100%, it seems to be averaging much closer to 95%.

However…
We also only have about 7,000 users connected across the cluster. It will be 
interesting to see what happens as the day progresses and students wake up and 
migrate from the ResNET cluster to the Campus cluster.
I’m also noticing that there are much fewer clients on this controller, and 
that ratio doesn’t seem to be improving.

I’ll update as we progress through this.


JD

--

Jonathan Davis

Wireless Architect

The University of North Carolina at Chapel Hill

jonath...@unc.edu<mailto:jonath...@unc.edu>

+1 336 279 3355 (Mobile)


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Lee H Badman 

Sent: Thursday, September 2, 2021 9:06:33 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

But you tested in your lab, right? I love that one… put new code on a couple of 
APs, or even a few dozen. That’s supposed to somehow indicate what will happen 
at bigger load… and also maybe implies the vendor didn’t do their own “similar 
lab testing”…

“You should have tested before upgrading the whole environment…” how do you 
REALLY do that? And should you really have to? Just pondering the general state 
of things.

> On Sep 2, 2021, at 08:59, Enfield, Chuck  wrote:
>
> That's been my experience for years.  The network works great when there are 
> no students around.  My working theory is that students emit RF interference, 
> but research ethics won’t let me run the tests, so we'll never know for sure.
>
> -Original Message-
> From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Patrick McEvilly
> Sent: Thursday, September 2, 2021 8:56 AM
> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
> any issues in the fall with large classrooms and delayed connection times 
> (Aruba 8.5.0.13)
>
> Speaking from experience, I would be very concerned.  We had no issues until 
> students returned and we went downhill from there.
>
>
> On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv 
> on behalf of Rob Harris"  robert.har...@culinary.edu> wrote:
>
>Has anyone seen any details regarding what they consider "Large" 
> environments? We upgraded during the break, but both before and after 
> versions are affected. We didn't notice this happening before, should we be 
> concerned now?
>
>The "dropped" is 0 and the stm cpu usage is in single digits, but client 
> count is really low (they come back this weekend as well), could we be in the 
> clear?
>
>(asked the SE team and opened a tac call, same questions to them)
>
>thx
>
>-Original Message-
>From: The EDUCAUSE Wireless Issues Community Group Listserv 
>  On Behalf Of Jason Healy
>    Sent: Thursday, September 2, 2021 8:45 AM
>    To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
> seeing any issues in the fall with large classrooms and delayed connection 
> times (Aruba 8.5.0.13)
>
>CAUTION: This email originated from outside The Culinary Institute of 
> America. Do not click links or open attachments unless you recognize the 
> sender and know the content is safe.
>
>FWIW, Aruba just posted an advisory regarding this issue:
>
>Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client 
> Connectivity Failures in Large Client Environments"
>
>Good luck to those of you hit by this. My students start coming back this 
> weekend so I'll be watching this closely!
>
>Jason
>**
>Replies to EDUCAUSE Community Group emails are sent to the entire 
> community list. If you want to reply only to the person who sent the message, 
> copy and paste their email address and forward the email reply. Additional 
> participation and subscription information can be found at 
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Michael Davis]

We see about 28K devices peak and so far haven't seen the issue crop 
up.  We did
just upgrade to 8.7.1.4 just before the Semester begun, because we were 
getting

smacked by the false radar detection bug that was causing AP reboots.

On 9/2/21 8:50 AM, Rob Harris wrote:

Has anyone seen any details regarding what they consider "Large" environments? 
We upgraded during the break, but both before and after versions are affected. We didn't 
notice this happening before, should we be concerned now?

The "dropped" is 0 and the stm cpu usage is in single digits, but client count 
is really low (they come back this weekend as well), could we be in the clear?

(asked the SE team and opened a tac call, same questions to them)

thx

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

CAUTION: This email originated from outside The Culinary Institute of America. 
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity Failures 
in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you 
want to reply only to the person who sent the message, copy and paste their email address 
and forward the email reply. Additional participation and subscription information can be 
found at 
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community



--
 Mike Davis
 IT - University of Delaware - 302.831.8756
 Newark, DE 19716   Email da...@udel.edu

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

When we had the problem last year, we couldn't see the high CPU usage much of 
the time.  The best way to tell if it's crashing is to look at the service 
uptime.  If it's been up for days or weeks you probably don't have the problem. 
 Hours, then you probably do.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Rob Harris
Sent: Thursday, September 2, 2021 8:50 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Has anyone seen any details regarding what they consider "Large" environments? 
We upgraded during the break, but both before and after versions are affected. 
We didn't notice this happening before, should we be concerned now?

The "dropped" is 0 and the stm cpu usage is in single digits, but client count 
is really low (they come back this weekend as well), could we be in the clear?

(asked the SE team and opened a tac call, same questions to them)

thx

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

CAUTION: This email originated from outside The Culinary Institute of America. 
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity 
Failures in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7Ce4da0822a6a143b7424d08d96e104391%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661838403488619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=L8kDlM2Vy7try2q1QdRgBCpOJKQiKGDTkUyY8%2FevgLU%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7Ce4da0822a6a143b7424d08d96e104391%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661838403488619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=L8kDlM2Vy7try2q1QdRgBCpOJKQiKGDTkUyY8%2FevgLU%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

That's been my experience for years.  The network works great when there are no 
students around.  My working theory is that students emit RF interference, but 
research ethics won’t let me run the tests, so we'll never know for sure.

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Patrick McEvilly
Sent: Thursday, September 2, 2021 8:56 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Speaking from experience, I would be very concerned.  We had no issues until 
students returned and we went downhill from there.


On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Rob Harris"  wrote:

Has anyone seen any details regarding what they consider "Large" 
environments? We upgraded during the break, but both before and after versions 
are affected. We didn't notice this happening before, should we be concerned 
now?

The "dropped" is 0 and the stm cpu usage is in single digits, but client 
count is really low (they come back this weekend as well), could we be in the 
clear?

(asked the SE team and opened a tac call, same questions to them)

thx

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
    Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

CAUTION: This email originated from outside The Culinary Institute of 
America. Do not click links or open attachments unless you recognize the sender 
and know the content is safe.

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity 
Failures in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Patrick McEvilly]

Speaking from experience, I would be very concerned.  We had no issues until 
students returned and we went downhill from there.


On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on 
behalf of Rob Harris"  wrote:

Has anyone seen any details regarding what they consider "Large" 
environments? We upgraded during the break, but both before and after versions 
are affected. We didn't notice this happening before, should we be concerned 
now?

The "dropped" is 0 and the stm cpu usage is in single digits, but client 
count is really low (they come back this weekend as well), could we be in the 
clear?

(asked the SE team and opened a tac call, same questions to them)

thx

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
    Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else 
seeing any issues in the fall with large classrooms and delayed connection 
times (Aruba 8.5.0.13)

CAUTION: This email originated from outside The Culinary Institute of 
America. Do not click links or open attachments unless you recognize the sender 
and know the content is safe.

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity 
Failures in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Rob Harris]

Has anyone seen any details regarding what they consider "Large" environments? 
We upgraded during the break, but both before and after versions are affected. 
We didn't notice this happening before, should we be concerned now?

The "dropped" is 0 and the stm cpu usage is in single digits, but client count 
is really low (they come back this weekend as well), could we be in the clear?

(asked the SE team and opened a tac call, same questions to them)

thx

-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

CAUTION: This email originated from outside The Culinary Institute of America. 
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity 
Failures in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Jason Healy]

FWIW, Aruba just posted an advisory regarding this issue:

Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity 
Failures in Large Client Environments"

Good luck to those of you hit by this. My students start coming back this 
weekend so I'll be watching this closely!

Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Patrick McEvilly]

Just a quick update that might be good information for others to have. We were 
running 8.7.1.3 since June 10th with no issues until yesterday.  We were seeing 
STM crash across several controllers and caused a significant outage in our 
wireless service.  We did the same things others here were told to do, remove 
SNMP, amp, add the FW rule etc.  We did an emergency upgrade to 8.7.1.4 today 
to address a bug that caused STM to crash.  We ran into another issue after the 
upgrade to 8.7.1.4 that would not allow about 50% of our APs to rejoin the 
controllers.  It is unknown if the issue was introduced in the new code or if 
it was pre-existing (unlikely pre-existing as we had to do a full controller 
reload last night and all the APs came back just fine). Over the course of the 
last 4-5 hours the remaining APs are still trickling in but we are not at 100%.

TAC are analyzing everything they captured on our marathon Zoom session.  For 
now, we are in a holding pattern and waiting for direction on how to proceed.  
If we make any breakthroughs we will be sure to share it here.

Patrick



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of "Enfield, Chuck" 

Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 

Date: Wednesday, September 1, 2021 at 5:29 PM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

We feel your pain, Patrick!  Keep up the good fight.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Patrick McEvilly
Sent: Wednesday, September 1, 2021 5:25 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

I will hold off on providing details for now but when you have to push a code 
upgrade in the middle of the day on the first day class it’s been a rough day.  
We hit some major issues related to STM and then other fall out after doing the 
required code upgrade.  We pushed the changes below at 2am this morning.  It 
did help a bit, but issues resurfaced again at 10am.  We are still on a call 
with Aruba TAC and don’t have anything at this time to share that would help 
others.

Patrick



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Date: Wednesday, September 1, 2021 at 5:13 PM
To: 
"WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

To all chiming in regarding the Aruba issues – thank you! I love seeing the 
collaboration and detail sharing.

Chad – will be curious to hear if you push the band-aids to production and 
re-enable airwaves, if this helps your situation.

-Cody


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 3:01 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>
__

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[McClintic, Thomas]

We had disabled "Prohibit ARP Spoofing" at one point to appease the 14+ code. 
The issue was resolved in a later release and we enabled it again.

We are not currently seeing any issues 8.5.0.11, moving to 13 now.

Thanks

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Enfield, Chuck
Sent: Wednesday, September 1, 2021 4:29 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


**** EXTERNAL EMAIL 
We feel your pain, Patrick!  Keep up the good fight.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Patrick McEvilly
Sent: Wednesday, September 1, 2021 5:25 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

I will hold off on providing details for now but when you have to push a code 
upgrade in the middle of the day on the first day class it's been a rough day.  
We hit some major issues related to STM and then other fall out after doing the 
required code upgrade.  We pushed the changes below at 2am this morning.  It 
did help a bit, but issues resurfaced again at 10am.  We are still on a call 
with Aruba TAC and don't have anything at this time to share that would help 
others.

Patrick



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Date: Wednesday, September 1, 2021 at 5:13 PM
To: 
"WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

To all chiming in regarding the Aruba issues - thank you! I love seeing the 
collaboration and detail sharing.

Chad - will be curious to hear if you push the band-aids to production and 
re-enable airwaves, if this helps your situation.

-Cody


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 3:01 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

We feel your pain, Patrick!  Keep up the good fight.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Patrick McEvilly
Sent: Wednesday, September 1, 2021 5:25 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

I will hold off on providing details for now but when you have to push a code 
upgrade in the middle of the day on the first day class it's been a rough day.  
We hit some major issues related to STM and then other fall out after doing the 
required code upgrade.  We pushed the changes below at 2am this morning.  It 
did help a bit, but issues resurfaced again at 10am.  We are still on a call 
with Aruba TAC and don't have anything at this time to share that would help 
others.

Patrick



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Date: Wednesday, September 1, 2021 at 5:13 PM
To: 
"WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

To all chiming in regarding the Aruba issues - thank you! I love seeing the 
collaboration and detail sharing.

Chad - will be curious to hear if you push the band-aids to production and 
re-enable airwaves, if this helps your situation.

-Cody


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 3:01 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F56.E9F8F5D0]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.ED

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Norton, Thomas (Network Operations)]

Don't forgot the nasty Lenovo vantage software 


T.J. Norton
Wireless Network Architect
Network Operations

(434) 592-6552

[http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg]

Liberty University  |  Training Champions for Christ since



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Turner, Ryan H 

Sent: Wednesday, September 1, 2021 5:12 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Glad I brought this up.  Is it possible that Cisco environments have evaded 
this?  Seems as though the ARP flooding via iOS 14 would be something that 
would menace all the manufacturers.



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 5:01 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)





Cody and all...



We are also seeing STM spikes that are impacting associations.



We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.



The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).



fingers crossed



here is how to block the traffic:

cd /md/yourrootlocation

firewall-cp

 ipv4 deny any proto 6 ports 15260 15261 position 1

!



Chad

chad.str...@emory.edu<mailto:chad.str...@emory.edu>



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)



I’m hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has “identified the issue 
and is working on a fix.” I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We’re seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren’t happening). Our SE also said if you’re experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F54.94BB2180]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We’ve also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are ‘we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected’.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn’t changed much. 

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Patrick McEvilly]

I will hold off on providing details for now but when you have to push a code 
upgrade in the middle of the day on the first day class it’s been a rough day.  
We hit some major issues related to STM and then other fall out after doing the 
required code upgrade.  We pushed the changes below at 2am this morning.  It 
did help a bit, but issues resurfaced again at 10am.  We are still on a call 
with Aruba TAC and don’t have anything at this time to share that would help 
others.

Patrick


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Cody Ensanian 

Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv 

Date: Wednesday, September 1, 2021 at 5:13 PM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

To all chiming in regarding the Aruba issues – thank you! I love seeing the 
collaboration and detail sharing.

Chad – will be curious to hear if you push the band-aids to production and 
re-enable airwaves, if this helps your situation.

-Cody


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 3:01 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I’m hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has “identified the issue 
and is working on a fix.” I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We’re seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren’t happening). Our SE also said if you’re experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F56.585D43B0]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We’ve also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are ‘we come into the classroom, and for some folks it can take up 
to 5 m

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

I'm speculating a bit, but Aruba does a lot of stuff with ARP if features like 
bcast ARP suppression, convert bacst to unicast, and BC/MC optimization are 
enabled.  I assume Cisco has some similar features, but perhaps not all of 
them?  Or maybe one key feature is causing most of the trouble for Aruba.

I also know that some of the ARP processing Aruba does on the controllers helps 
reduce the amount of ARP that reaches the underpinning network.  I'm sure many 
of us have ARP policers kicking in right now.  I hate to think about what our 
switches and routers would be struggling with if the controllers didn't manage 
this stuff like they do.  We may have to pick our poison.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 5:13 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

Glad I brought this up.  Is it possible that Cisco environments have evaded 
this?  Seems as though the ARP flooding via iOS 14 would be something that 
would menace all the manufacturers.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 5:01 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F55.B5EE9F70]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We've also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are 'we come i

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Tolka, Bryan]

Ryan,

Do you have multicast enabled ?What is the mandatory rate you are using in 
the classrooms?

We just had some issues with this not on Aruba .

Bryan Tolka

Sent from my iPhone

On Sep 1, 2021, at 5:00 PM, Street, Chad A  wrote:



Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Cody Ensanian 

Sent: Wednesday, September 1, 2021 11:41 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I’m hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has “identified the issue 
and is working on a fix.” I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We’re seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren’t happening). Our SE also said if you’re experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F14.D04A94F0]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We’ve also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are ‘we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected’.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn’t changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.



Has anyone else seen something like this?



Ryan Turner

Head of Networking

Communication Technologies | Information Technology Services

r...@unc.edu<mailto:r...@unc.edu>

+1 919 445 0113 (Office)

+1 919 274 7926 (Mobile)



**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Cbtolka%40HSC.WVU.EDU%7Cdac0d4e893364f64b8a108d96d8b964f%7Ca2d1f95f851044248ae15c596bdbd578%7C0%7C0%7C637661268570757361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=tYz0OWkhXLFkiLNk3U2PeYLMMbT%2B8dAiUTa1gRmKaIg%3D=0>

**
Replies to EDUCAUSE Community Group ema

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Cody Ensanian]

To all chiming in regarding the Aruba issues - thank you! I love seeing the 
collaboration and detail sharing.

Chad - will be curious to hear if you push the band-aids to production and 
re-enable airwaves, if this helps your situation.

-Cody


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 3:01 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F43.704FEB70]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We've also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are 'we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected'.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn't changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.



Has anyone else seen something like this?



Ryan Turner

Head of Networking

Communication Technologies | Information Technology Services

r...@unc.edu<mailto:r...@unc.edu>

+1 919 445 0113 (Office)

+1 919 274 7926 (Mobile)



**
Replies to EDUCAUSE Community Group emails are sent to the entire communi

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Turner, Ryan H]

Glad I brought this up.  Is it possible that Cisco environments have evaded 
this?  Seems as though the ARP flooding via iOS 14 would be something that 
would menace all the manufacturers.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Street, Chad A
Sent: Wednesday, September 1, 2021 5:01 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)


Cody and all...

We are also seeing STM spikes that are impacting associations.

We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the 
client load balancing thresholds so that we have around 4K clients per 
controller.  This seemed to help a great deal.  After working with Aruba today, 
my understanding of the primary cause of the STM spikes is due to the MM 
polling the MCs.  With large client loads on the MCs ( combined with all the 
other SNMP polling going on ), this seems to take longer and sometimes does not 
work.  When it does not work, it bootstraps which spikes the STM process.

The suggested band-aid is to block the GUI polling traffic between the MM and 
MC.  You will lose the GUI information from your MM, but all the MC information 
is still present.  We have applied this to our lab and we are going to push to 
production tonight to see if it helps.  If it does help, we plan on turning 
back up our monitoring tools ( Airwave ).

fingers crossed

here is how to block the traffic:
cd /md/yourrootlocation
firewall-cp
 ipv4 deny any proto 6 ports 15260 15261 position 1
!

Chad
chad.str...@emory.edu<mailto:chad.str...@emory.edu>

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Cody Ensanian mailto:censa...@uccs.edu>>
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the 
fall with large classrooms and delayed connection times (Aruba 8.5.0.13)


I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F54.94BB2180]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We've also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are 'we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected'.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn't changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.



Has anyone else seen something like this?



Ryan Turner

Head of Networking

Communication Technologies | Information Technology Services

r...@unc.edu<mailto:r...@unc.edu>

+1 919 445 0113 (Office)

+1 919 274 7926 (Mobile)



**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent th

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

Please see my 12:05 response if you missed it.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Steve Smith
Sent: Wednesday, September 1, 2021 12:00 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

I wouldn't mind seeing the arp limiting client filter as well.

Thank you,
Steve

Steve Smith
Network Administrator II
Network and Telecommunications Services
Aims Community College
970.339.6565

On Wed, Sep 1, 2021 at 9:57 AM Laramie Combs 
mailto:comb...@appstate.edu>> wrote:
HEy Chuck - would you mind sharing that arp limiting client filter with me?

We are seeing some new traffic patterns where it looks like user devices are 
just walking their subnets, and arping for everything

-Laramie

On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck 
mailto:cae...@psu.edu>> wrote:
We've seen the CPU problem, but I don't think it resulted in Auth problems 
here.  It may have and we just missed it because the more severe problems it 
caused masked them.

BTW, in our case reducing the amount of ARP calmed the CPU.  We applied a 
filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless 
clients could send and it smoothed out the spikes.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Cody Ensanian
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with 
large classrooms and delayed connection times (Aruba 8.5.0.13)

I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image001.png@01D79F2A.AE293D20]

-Cody
UCCS


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)

This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We've also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are 'we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected'.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn't changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.

Has anyone else seen something like this?

Ryan Turner
Head of Networking
Communication Technologies | Information Technology Services
r...@unc.edu<mailto:r...@unc.edu>
+1 919 445 0113 (Office)
+1 919 274 7926 (Mobile)


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__nam10.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.educause.edu-252Fcommunity-26data-3D04-257C01-257Ccae104-2540PSU.EDU-257C23c7b4692be5427984b208d96d5eeb66-257C7cf48d453ddb4389a9c1c115

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Steve Smith]

I wouldn't mind seeing the arp limiting client filter as well.

Thank you,
Steve

Steve Smith
Network Administrator II
Network and Telecommunications Services
Aims Community College
970.339.6565

On Wed, Sep 1, 2021 at 9:57 AM Laramie Combs  wrote:

> HEy Chuck - would you mind sharing that arp limiting client filter with me?
>
> We are seeing some new traffic patterns where it looks like user devices
> are just walking their subnets, and arping for everything
>
> -Laramie
>
> On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck  wrote:
>
>> We’ve seen the CPU problem, but I don’t think it resulted in Auth
>> problems here.  It may have and we just missed it because the more severe
>> problems it caused masked them.
>>
>>
>>
>> BTW, in our case reducing the amount of ARP calmed the CPU.  We applied a
>> filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless
>> clients could send and it smoothed out the spikes.
>>
>>
>>
>> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
>> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Cody Ensanian
>> *Sent:* Wednesday, September 1, 2021 11:41 AM
>> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>> *Subject:* Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall
>> with large classrooms and delayed connection times (Aruba 8.5.0.13)
>>
>>
>>
>> I’m hearing issues of high cpu utilization for STM on the controllers
>> causing issues. Maybe check your controllers and see if you are seeing the
>> high cpu use for STM. Heard earlier today from our SE that Aruba has
>> “identified the issue and is working on a fix.” I suggest opening the TAC
>> case so they can track it better, and help them hone in on a fix better.
>> We’re seeing the high cpu use on one of our controllers (but this
>> controller also has higher client load). However, we have not had a flood
>> of calls to our help desk for wireless issues (not saying they aren’t
>> happening). Our SE also said if you’re experiencing the issue, disabling
>> any system or process level debugging as helped, as well as disabling any
>> SNMP polling.
>>
>>
>>
>> -Cody
>>
>> UCCS
>>
>>
>>
>>
>>
>> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
>> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Turner, Ryan H
>> *Sent:* Wednesday, September 1, 2021 9:27 AM
>> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>> *Subject:* [WIRELESS-LAN] Anyone else seeing any issues in the fall with
>> large classrooms and delayed connection times (Aruba 8.5.0.13)
>>
>>
>>
>> This is a stab in the dark.  With the University mostly shutdown since
>> the Spring of 2020 (=not operating in standard mode and most people work
>> from home), we got campus upgraded from 6.X to 8.X code base.  We’ve also
>> installed many 515 series APs.  We are getting a large number of complaints
>> in large classrooms that connecting to things like eduroam takes a long
>> time.  Looking into the connection, we see many incomplete RADIUS
>> challenges.  The general complaints are ‘we come into the classroom, and
>> for some folks it can take up to 5 minutes to get connected’.  The odd
>> thing is that our RADIUS infrastructure is very large, polished and load
>> shared, and we can see no performance issues with any of the RADIUS
>> servers.  We have begun reducing power in the large classrooms to make
>> association issues better, but so far that hasn’t changed much.  We
>> anticipate opening a ticket with Aruba, soon.  We do seem to see the most
>> complaints in the big classrooms.  But I do keep going back to the RADIUS
>> Challenges incomplete.  I know if no reason for those not to complete
>> unless the connection is broken midway.
>>
>>
>>
>> Has anyone else seen something like this?
>>
>>
>>
>> Ryan Turner
>>
>> Head of Networking
>>
>> Communication Technologies | Information Technology Services
>>
>> r...@unc.edu
>>
>> +1 919 445 0113 (Office)
>>
>> +1 919 274 7926 (Mobile)
>>
>>
>>
>> **
>> Replies to EDUCAUSE Community Group emails are sent to the entire
>> community list. If you want to reply only to the person who sent the
>> message, copy and paste their email address and forward the email reply.
>> Additional participation and subscription information can be found at
>> https://www.educause.edu/community
>> 
>>
>> **
>> Replies to EDUCAUSE Community Group 

Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Norton, Thomas (Network Operations)]

Hey Laramine/Chuck,

The ARP issue most likely the Lenovo Vantage software or IOS 14. Another option 
outside of filtering is to enable prohibit ip spoofing and arp spoofing.



T.J. Norton
Wireless Network Architect
Network Operations

(434) 592-6552

[http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg]

Liberty University  |  Training Champions for Christ since



From: The EDUCAUSE Wireless Issues Community Group Listserv 
 on behalf of Laramie Combs 

Sent: Wednesday, September 1, 2021 11:57 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU 
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

HEy Chuck - would you mind sharing that arp limiting client filter with me?

We are seeing some new traffic patterns where it looks like user devices are 
just walking their subnets, and arping for everything

-Laramie

On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck 
mailto:cae...@psu.edu>> wrote:

We’ve seen the CPU problem, but I don’t think it resulted in Auth problems 
here.  It may have and we just missed it because the more severe problems it 
caused masked them.



BTW, in our case reducing the amount of ARP calmed the CPU.  We applied a 
filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless 
clients could send and it smoothed out the spikes.



From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Cody Ensanian
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with 
large classrooms and delayed connection times (Aruba 8.5.0.13)



I’m hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has “identified the issue 
and is working on a fix.” I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We’re seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren’t happening). Our SE also said if you’re experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:17ba213cca04cff311]



-Cody

UCCS





From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)



This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We’ve also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are ‘we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected’.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn’t changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.



Has anyone else seen something like this?



Ryan Turner

Head of Networking

Communication Technologies | Information Technology Services

r...@unc.edu<mailto:r...@unc.edu>

+1 919 445 0113 (Office)

+1 919 274 7926 (Mobile)



**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C7bfb23fbcabf4dbaa9eb08d96d614285%7Cbaf8218eb3024465a9934a39c97251b2

RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)

[Enfield, Chuck]

Here's what we did.  We had a pcap that suggested only about 1% of clients 
would be affected by this filter, but it cut our ARP almost in half.  We made 
the change last spring in our res halls which were almost fully occupied, and 
we've not traced user complaints back to this yet.

[cid:image001.png@01D79F29.7E132B60]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Laramie Combs
Sent: Wednesday, September 1, 2021 11:58 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing 
any issues in the fall with large classrooms and delayed connection times 
(Aruba 8.5.0.13)

HEy Chuck - would you mind sharing that arp limiting client filter with me?

We are seeing some new traffic patterns where it looks like user devices are 
just walking their subnets, and arping for everything

-Laramie

On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck 
mailto:cae...@psu.edu>> wrote:
We've seen the CPU problem, but I don't think it resulted in Auth problems 
here.  It may have and we just missed it because the more severe problems it 
caused masked them.

BTW, in our case reducing the amount of ARP calmed the CPU.  We applied a 
filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless 
clients could send and it smoothed out the spikes.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Cody Ensanian
Sent: Wednesday, September 1, 2021 11:41 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with 
large classrooms and delayed connection times (Aruba 8.5.0.13)

I'm hearing issues of high cpu utilization for STM on the controllers causing 
issues. Maybe check your controllers and see if you are seeing the high cpu use 
for STM. Heard earlier today from our SE that Aruba has "identified the issue 
and is working on a fix." I suggest opening the TAC case so they can track it 
better, and help them hone in on a fix better. We're seeing the high cpu use on 
one of our controllers (but this controller also has higher client load). 
However, we have not had a flood of calls to our help desk for wireless issues 
(not saying they aren't happening). Our SE also said if you're experiencing the 
issue, disabling any system or process level debugging as helped, as well as 
disabling any SNMP polling.

[cid:image002.png@01D79F29.7E132B60]

-Cody
UCCS


From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Turner, Ryan H
Sent: Wednesday, September 1, 2021 9:27 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large 
classrooms and delayed connection times (Aruba 8.5.0.13)

This is a stab in the dark.  With the University mostly shutdown since the 
Spring of 2020 (=not operating in standard mode and most people work from 
home), we got campus upgraded from 6.X to 8.X code base.  We've also installed 
many 515 series APs.  We are getting a large number of complaints in large 
classrooms that connecting to things like eduroam takes a long time.  Looking 
into the connection, we see many incomplete RADIUS challenges.  The general 
complaints are 'we come into the classroom, and for some folks it can take up 
to 5 minutes to get connected'.  The odd thing is that our RADIUS 
infrastructure is very large, polished and load shared, and we can see no 
performance issues with any of the RADIUS servers.  We have begun reducing 
power in the large classrooms to make association issues better, but so far 
that hasn't changed much.  We anticipate opening a ticket with Aruba, soon.  We 
do seem to see the most complaints in the big classrooms.  But I do keep going 
back to the RADIUS Challenges incomplete.  I know if no reason for those not to 
complete unless the connection is broken midway.

Has anyone else seen something like this?

Ryan Turner
Head of Networking
Communication Technologies | Information Technology Services
r...@unc.edu<mailto:r...@unc.edu>
+1 919 445 0113 (Office)
+1 919 274 7926 (Mobile)


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7Cbdf812520057426b3d3f08d96d614254%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661086771803279%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=CMis8