Re: [WIRELESS-LAN] Aruba information sharing Zoom call
On Sep 17, 2021, at 21:29, Patrick McEvilly mailto:patrick_mcevi...@harvard.edu>> wrote: This group has been a great help to us as we dealt with several issues over the past two weeks related to our Aruba wireless infrastructure. Just to add our experiences to the mix…..Aruba came out with a more widely available 8.7.1.5 C build last week. We installed it on our beta/test controller cluster on Friday and are planning on doing one of our campus controller clusters tomorrow morning and then the other on Wednesday. The other published mitigations have had us in a stable state over the past 10 days or so. Then we are all waiting for the 8.7.1.6 GA build in a couple of weeks. Fingers crossed…..thanks to everyone here and at Aruba who has been sharing very helpful information and working with us through all of this. -- Julian Y. Koh Director, Telecommunications and Network Services Northwestern Information Technology 2020 Ridge Avenue #331 Evanston, IL 60208 +1-847-467-5780 Northwestern IT Web Site: <https://www.it.northwestern.edu/> PGP Public Key: <https://bt.ittns.northwestern.edu/julian/pgppubkey.html> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
What rates did you set for the ARP policing? Thanks Chris Hart [cid:image001.png@01D7A94B.455C09A0] Chris Hart Network Operations Engineer Lead Tel: 847-467-7747 Email: ch...@northwestern.edu<mailto:ch...@northwestern.edu> 2020 Ridge Ave, Evanston, IL From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Viou, Robert Sent: Saturday, September 11, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) After working with the Aruba TAC last night, these are the changes we made that appear to have corrected the issues we were seeing. Disabling Airgroup temporarily stops the issue with Queuing of the Arp packets. But the changes that we added below allowed us to re-enable Airgroup with APGroup set in the Profile. Still need to monitor to be sure it is fixed, but so far looks good. Monitor/police non-gratuitous ARP attacks: ENABLED Monitor/police non-gratuitous ARP attack action: DROP Bob From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 9:12 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://urldefense.com/v3/__https:/nam02.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.facebook.com*2FISUITHelp*2F=04*7C01*7CRobert.Viou*40NDSU.EDU*7Cdd0c720e506c47cda65308d9752e34d8*7Cec37a091b9a647e598d0903d4a419203*7C1*7C0*7C637669663628347496*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000=TP7NNp8n1*2BVyS2hYfqa7cYLY0bjswlO0FqAqTKioBQk*3D=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrRjZULnc$> and Twitter<https://urldefense.com/v3/__https:/nam02.safelinks.protection.outlook.com/?url=https*3A*2F*2Ftwitter.com*2FISUITHelp=04*7C01*7CRobert.Viou*40NDSU.EDU*7Cdd0c720e506c47cda65308d9752e34d8*7Cec37a091b9a647e598d0903d4a419203*7C1*7C0*7C637669663628357488*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000=RMTjQdg9p3bfKvhQcn*2BylQWZg2I*2FI3MyRPn31Qnh5rs*3D=0__;JSUlJSUlJSUlJSUlJSUlJSUlJQ!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrg4Vpb6Q$> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Some people who received this message don't often get email from robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is important<https://urldefense.com/v3/__http:/aka.ms/LearnAboutSenderIdentification__;!!Dq0X2DkFhyF93HkjWTBQKhk!B0zi5_9S-YJgbcLXO3V2Gp9eodGAJkAvcl9Yf_7gjJ4zM_PVEU4Txe1S-wPrBMxB_YE$> [This message came from an external source. If suspicious, report to ab...@ilstu.
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Hello All, I wanted to give you an update. First, I’ll provide a -We (UNC-CH) seem to be in a stable situation. Clients are connecting and staying connected. STM is consuming reasonable levels of resources. -We are blind. In addition to disabling the MM to MC communication, we had to disable all SNMP. We also have a bit of automation that utilizes SNMP, and that is also broken. -There were TWO separate issues, and I’ll break those out below. It’s important to understand that the ARP issue is a separate issue from the STM issue. In our first couple of days working with Aruba, even TAC did not recognize our symptoms as two separate issues, and this is probably the thing that frustrated me the most. Issue number one: ARP Our users complained they could not connect to the network and access resources for the first five to ten minutes of a class. Apple IOS devices running version 14, and some Lenovo devices ARP their entire subnet after joining Wi-Fi. The Aruba controllers have security rules in place to prevent ARP flooding and DDOS attacks utilizing ARP. As clients joined the network before classes the devices would ARP the subnet, and once a threshold was reached, the controller would begin discarding ARP packets for all clients on that controller. The result was that devices would connect, get assigned an IP via DHCP, and then ARP to get the MAC of their default gateway. That packet would be discarded, and until the controller again allowed ARP to pass, clients weren’t able to find their gateway. Depending on the client, this usually resulted in them again restarting the 802.11 join process. [Christopher Johnson, this is the behavior you are experiencing.] You can see if you are being affected by running: show datapath bwm table and checking for contract 9 (ARP). You can also check this more specifically by running: show datapath bwm type [type] contract 9 In our case, the full command was: show datapath bwm type 0 contract 9 When we first addressed this issue, we had over 2 million drops (policed) packets on each controller. Our default configuration was 992pps. After consideration, we raised our rate to 9792 expecting that multiple clients will likely be ARPing the network at the same time and recognizing how large the subnet is… and hey, it seemed like a good idea. Since then, we average less than 1-3K drops at any given time, and our users are telling us they can connect and access the network on the first try. We have seen no other detrimental effects of this change. NEXT – STM We disabled our connections between the MM and MC’s and restarted all controllers by controller cluster groups to ensure AP’s and Clients would stay connected. Once everything was restarted, we waited for students to migrate from ResNET to our Main Campus cluster. We began getting the first complaints around 10am. After checking load distribution, we found that we had even distribution of AP’s across our 8 MC’s, but 90% of our clients were connected to only two of our eight controllers in that cluster despite our load balancing configuration. This continued to be an issue, and TAC confirmed that we were appropriately configured to load balance clients at 10%. Despite disabling the MM to MC connections, we still had very high utilization by STM, and TAC decided controllers were unable to balance client connections due to that state. The next step was to block SNMP on the controller firewalls. As you can all imagine, this was a difficult decision for us, but if clients can’t connect to Wi-Fi, we don’t need SNMP to tell us it’s down…the users do a great job of that! Once we disabled SNMP, STM processor usage fell to ~30-70% and clients began balancing appropriately across controllers. So, as I said in my TLDR, we are flying blind, but user reports are coming in that the issue is much improved. Now we wait for Aruba to deliver our bug fix, and a bit of time for testing to ensure we don’t cause more issues. I want to pause here and express my second large frustration with the situation. For the affected cluster, we are running eight 7240XM controllers, which according to Aruba should support 32K clients each, yet those two controllers were incapable of load balancing due to high STM utilization when each had only 8K clients. Like many who have spoken up, we begin seeing issues as soon as client counts on a controller exceed 5K clients. I shudder to think what our experience would have been if we had half as many controllers in the cluster. Marketecture != good design JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of James Andrewartha Date: Saturday, September 11, 2021 at 9:49 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in th
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I'm not too familiar with how Aruba handles arps, does it do proxy arp? I have seen Apple devices go to sleep before all broadcast/multicast traffic is sent by the AP, although that was 5 years ago. So I can believe that a behaviour change could cause increased ARPs if the devices aren't seeing them. Sent from my Galaxy Original message From: "Turner, Ryan H" Date: 12/9/21 09:16 (GMT+08:00) To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We actually are allowing MORE ARPs. Apparently when policing kicks in, all connections are affecting. It can cause clients to freeze/not connect. So we actually turned the knob in the opposite direction. We were seeing counters to what amounts to large quantities of controllers pauses when the ARPs went over an arbitrarily set number. Our wireless architect can reply with the details. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 11, 2021, at 12:32 PM, Enfield, Chuck wrote: HI Ryan, When you say that you detuned ARP policing, do you mean that the ARP policing on the underpinning network is now more aggressive (aka, dropping more ARP?) I ask because I’ve been wondering why we aren’t seeing this problem when other schools that made the same changes we did still are. We upgraded our underpinning network over the summer, and we’re dropping way more ARP than we were on the old network. Your post just made me realize that may be protecting our controllers. We’ve been considering changes, but we switched to an EVPN/VxLAN architecture. We’re not completely sure what the consequences of this ARP policing is, so we’ve been holding off any changes. If you had to police more aggressively to solve your problem, then we won’t start experimenting with out policers. Thanks, Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 10:12 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0> and Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] An
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We actually are allowing MORE ARPs. Apparently when policing kicks in, all connections are affecting. It can cause clients to freeze/not connect. So we actually turned the knob in the opposite direction. We were seeing counters to what amounts to large quantities of controllers pauses when the ARPs went over an arbitrarily set number. Our wireless architect can reply with the details. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 11, 2021, at 12:32 PM, Enfield, Chuck wrote: HI Ryan, When you say that you detuned ARP policing, do you mean that the ARP policing on the underpinning network is now more aggressive (aka, dropping more ARP?) I ask because I’ve been wondering why we aren’t seeing this problem when other schools that made the same changes we did still are. We upgraded our underpinning network over the summer, and we’re dropping way more ARP than we were on the old network. Your post just made me realize that may be protecting our controllers. We’ve been considering changes, but we switched to an EVPN/VxLAN architecture. We’re not completely sure what the consequences of this ARP policing is, so we’ve been holding off any changes. If you had to police more aggressively to solve your problem, then we won’t start experimenting with out policers. Thanks, Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 10:12 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0> and Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Some people who received this message don't often get email from robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] In regards to: > Aruba believes this is the cause of the new iOS operating system. Our > environment is extremely heavy iOS. We are talking to them now and will > assess the change. Has Aruba menti
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
After working with the Aruba TAC last night, these are the changes we made that appear to have corrected the issues we were seeing. Disabling Airgroup temporarily stops the issue with Queuing of the Arp packets. But the changes that we added below allowed us to re-enable Airgroup with APGroup set in the Profile. Still need to monitor to be sure it is fixed, but so far looks good. Monitor/police non-gratuitous ARP attacks: ENABLED Monitor/police non-gratuitous ARP attack action: DROP Bob From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 9:12 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7CRobert.Viou%40NDSU.EDU%7Cdd0c720e506c47cda65308d9752e34d8%7Cec37a091b9a647e598d0903d4a419203%7C1%7C0%7C637669663628347496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=TP7NNp8n1%2BVyS2hYfqa7cYLY0bjswlO0FqAqTKioBQk%3D=0> and Twitter<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7CRobert.Viou%40NDSU.EDU%7Cdd0c720e506c47cda65308d9752e34d8%7Cec37a091b9a647e598d0903d4a419203%7C1%7C0%7C637669663628357488%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=RMTjQdg9p3bfKvhQcn%2BylQWZg2I%2FI3MyRPn31Qnh5rs%3D=0> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Some people who received this message don't often get email from robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] In regards to: > Aruba believes this is the cause of the new iOS operating system. Our > environment is extremely heavy iOS. We are talking to them now and will > assess the change. Has Aruba mentioned to anyone or has anyone heard any more on this being an IOS issue? Bob From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 11:31 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thanks for the update Ryan T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-L
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
HI Ryan, When you say that you detuned ARP policing, do you mean that the ARP policing on the underpinning network is now more aggressive (aka, dropping more ARP?) I ask because I’ve been wondering why we aren’t seeing this problem when other schools that made the same changes we did still are. We upgraded our underpinning network over the summer, and we’re dropping way more ARP than we were on the old network. Your post just made me realize that may be protecting our controllers. We’ve been considering changes, but we switched to an EVPN/VxLAN architecture. We’re not completely sure what the consequences of this ARP policing is, so we’ve been holding off any changes. If you had to police more aggressively to solve your problem, then we won’t start experimenting with out policers. Thanks, Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Saturday, September 11, 2021 10:12 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We had to make major changes to bring stability to Khrushchev environment. I think we have at this point. We had to significantly detune the ARP policing policies. We had to block virtually every SNMP poller. We had to reboot our controllers. We had to put in place an ACL to block communication from the Mobility masters. A ridiculous amount of work to basically get us where we were 2 years ago and we probably have 15% lower connections compared to then. I am hoping that the upcoming firmware fix will allow us to at least reverse the ACL and SNMP pollers. At this point we are pretty blind into information on individual connections. Ryan Turner Head of Networking, ITS The University of North Carolina at Chapel Hill +1 919 274 7926 Mobile +1 919 445 0113 Office On Sep 10, 2021, at 4:25 PM, Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: I haven’t heard anything as of yet. Although interestingly while doing a packet-capture to monitor arp/dhcp rates – noticed one client sending DHCPRequests about 3-4-5 times a minutes – and disassociating/re-associating constantly – and from the received signal strength of the client – there didn’t appear to be any reason for this iPhone – 14.7.X – to behave in such a matter. So I’m wondering if that’s not an isolated behavior. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621408058%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=BGYNDoJo6qpi83pjd9pMIP7EO9lv0sl4L4S4AkNKLfk%3D=0> and Twitter<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Ccae104%40PSU.EDU%7C3e82d4d6e9524db4270a08d9752e3652%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637669663621418052%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=wl8cYjmkPpNkBgtwT7j6IlVr0mOlFkLMIKo6knZ82fM%3D=0> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Viou, Robert Sent: Friday, September 10, 2021 10:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Some people who received this message don't often get email from robert.v...@ndsu.edu<mailto:robert.v...@ndsu.edu>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] In regards to: > Aruba believes this is the cause of the new iOS operating system. Our > environment is extremely heavy iOS. We are talking to them now and will > assess the change. Has Aruba mentioned to anyone or has anyone heard any more on this being an IOS issue? Bob From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 11:31 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Thanks for the update Ryan T.J. Norton
RE: [WIRELESS-LAN] Aruba 515 IAP - High Efficiency Mode Question
Ron, Did you verify with certainty that you were connected to the AP that you thought you were on? -Lee Lee Badman | Network Architect (CWNE#200) Information Technology Services (NDD Group) 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 e lhbad...@syr.edu<mailto:lhbad...@syr.edu> w its.syr.edu Campus Wireless Policy: https://answers.syr.edu/display/network/Wireless+Network+and+Systems SYRACUSE UNIVERSITY syr.edu From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Ronald Loneker Sent: Friday, September 10, 2021 2:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Aruba 515 IAP - High Efficiency Mode Question Hi Everyone - This past spring we deployed several buildings with Aruba IAP 515 access points. This summer, we had the company who installed the access points produce heat mapping summaries of the buildings. In three of the four buildings, we had high efficiency mode enabled on the access points. Has anyone using these access points noticed a degraded signal when this mode is enabled? I was sitting almost in front of one of the access points that is showing to be active and pretty decent coveage where I was sitting but getting very low wireless signal from my laptop (even after I rebooted the laptop, disconnected from wifi and reconnected). I'm trying to get a version of the firmware we are running - there was a conflict in one of the buildings that had a cluster of 215s and 515s and the 215s couldn't run the more recent version of firmware so our consultant may have downgraded us to one that both models could support. Just curious about experiences you might have had with the high efficiency mode on and off and whether signal is better on either setting. Ron Loneker, Jr. Director, IT Special Projects Saint Elizabeth University Mahoney Library 2 Convent Road Morristown, NJ 07960 Phone: 973-290-4229 e-mail: rlone...@steu.edu<mailto:rlone...@steu.edu> Saint Elizabeth University's IT department will never ask for your password, social security number or other personal information in an e-mail message. Please do not share any information with others! ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] Aruba 515 IAP - High Efficiency Mode Question
Actually, Lee, the AP kept dropping me so much I couldn't even get a connection so I couldn't tell you. Being I was in a meeting and trying to participate, I couldn't do much testing. I'm doing tech support for an event all day so I can't pull out my floor plans to see if there was an access point on the floor above me - that would be the only other explanation.. Ron --- Ron Loneker, Jr. Director, IT Special Projects Saint Elizabeth University Mahoney Library 2 Convent Road Morristown, NJ 07960 Phone: 973-290-4229 e-mail: rlone...@steu.edu *Saint Elizabeth University's IT department will never ask for your password, social security number or other personal information in an e-mail message.* *Please do not share any information with others!* On Fri, Sep 10, 2021 at 3:00 PM Lee H Badman wrote: > Ron, > > > > Did you verify with certainty that you were connected to the AP that you > thought you were on? > > > > -Lee > > > > *Lee Badman* | Network Architect (CWNE#200) > > Information Technology Services > (NDD Group) > 206 Machinery Hall > 120 Smith Drive > Syracuse, New York 13244 > > *t* 315.443.3003 * e* lhbad...@syr.edu *w* its.syr.edu > > Campus Wireless Policy: > https://answers.syr.edu/display/network/Wireless+Network+and+Systems > > *SYRACUSE UNIVERSITY* > syr.edu > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Ronald Loneker > *Sent:* Friday, September 10, 2021 2:41 PM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* [WIRELESS-LAN] Aruba 515 IAP - High Efficiency Mode Question > > > > Hi Everyone - > > > > This past spring we deployed several buildings with Aruba IAP 515 access > points. > > > > This summer, we had the company who installed the access points produce > heat mapping summaries of the buildings. > > > > In three of the four buildings, we had high efficiency mode enabled on the > access points. > > > > Has anyone using these access points noticed a degraded signal when this > mode is enabled? I was sitting almost in front of one of the access points > that is showing to be active and pretty decent coveage where I was sitting > but getting very low wireless signal from my laptop (even after I rebooted > the laptop, disconnected from wifi and reconnected). > > > > I'm trying to get a version of the firmware we are running - there was a > conflict in one of the buildings that had a cluster of 215s and 515s and > the 215s couldn't run the more recent version of firmware so our consultant > may have downgraded us to one that both models could support. > > > > Just curious about experiences you might have had with the high efficiency > mode on and off and whether signal is better on either setting. > > > Ron Loneker, Jr. > Director, IT Special Projects > Saint Elizabeth University > Mahoney Library > 2 Convent Road > Morristown, NJ 07960 > > Phone: 973-290-4229 > > e-mail: rlone...@steu.edu > > > > *Saint Elizabeth University's IT department will never ask for your > password, social security number or other personal information in an e-mail > message. * > *Please do not share any information with others!* > > > > > > > > > > > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] Aruba 515 IAP - High Efficiency Mode Question
Hi Ron, We moved from AOS6 to AOS8 over the summer. Upon students' return we noticed what we felt to be degraded signal strength and degraded device performance. We will be disabling this HE setting on all our radio profiles in the coming days. We had confirmation for our Aruba SE and from a peer college that we should go ahead and disable this setting. Thanks, Dan On Fri, Sep 10, 2021 at 2:41 PM Ronald Loneker wrote: > Hi Everyone - > > This past spring we deployed several buildings with Aruba IAP 515 access > points. > > This summer, we had the company who installed the access points produce > heat mapping summaries of the buildings. > > In three of the four buildings, we had high efficiency mode enabled on the > access points. > > Has anyone using these access points noticed a degraded signal when this > mode is enabled? I was sitting almost in front of one of the access points > that is showing to be active and pretty decent coveage where I was sitting > but getting very low wireless signal from my laptop (even after I rebooted > the laptop, disconnected from wifi and reconnected). > > I'm trying to get a version of the firmware we are running - there was a > conflict in one of the buildings that had a cluster of 215s and 515s and > the 215s couldn't run the more recent version of firmware so our consultant > may have downgraded us to one that both models could support. > > Just curious about experiences you might have had with the high efficiency > mode on and off and whether signal is better on either setting. > > Ron Loneker, Jr. > Director, IT Special Projects > Saint Elizabeth University > Mahoney Library > 2 Convent Road > Morristown, NJ 07960 > > Phone: 973-290-4229 > > e-mail: rlone...@steu.edu > > > > *Saint Elizabeth University's IT department will never ask for your > password, social security number or other personal information in an e-mail > message.* > *Please do not share any information with others!* > > > > > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > -- *Daniel Wurst* Network Engineer II* | *Information Technology Services Denison University | 100 West College Street, Granville, OH 43023 | Burton Hall 740-587-6229 | wur...@denison.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba 515 IAP - High Efficiency Mode Question
Hi Everyone - This past spring we deployed several buildings with Aruba IAP 515 access points. This summer, we had the company who installed the access points produce heat mapping summaries of the buildings. In three of the four buildings, we had high efficiency mode enabled on the access points. Has anyone using these access points noticed a degraded signal when this mode is enabled? I was sitting almost in front of one of the access points that is showing to be active and pretty decent coveage where I was sitting but getting very low wireless signal from my laptop (even after I rebooted the laptop, disconnected from wifi and reconnected). I'm trying to get a version of the firmware we are running - there was a conflict in one of the buildings that had a cluster of 215s and 515s and the 215s couldn't run the more recent version of firmware so our consultant may have downgraded us to one that both models could support. Just curious about experiences you might have had with the high efficiency mode on and off and whether signal is better on either setting. Ron Loneker, Jr. Director, IT Special Projects Saint Elizabeth University Mahoney Library 2 Convent Road Morristown, NJ 07960 Phone: 973-290-4229 e-mail: rlone...@steu.edu *Saint Elizabeth University's IT department will never ask for your password, social security number or other personal information in an e-mail message.* *Please do not share any information with others!* ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
Have seen similar behavior and strongly recommend using validuser acl at very least change it form default any any- can start small and deny/protect critical IP's in your infrastructure its all fun and games until a user device gets picked up as your DNS server or local ip gateway but would recommend ultimately making validuser acl only accept ip's you expect your client to have when it's happening it sure seems malicious - but have learned not to assign intent to most actions of my users. On Tue, Sep 7, 2021 at 12:53 PM Johnson, Christopher wrote: > Sid, > > > We know from personal experience of running into this issue several years > ago. Like David, we’ve instituted a few validuserACLs – (I actually use > aliases for those subnets – so that I can re-use them in other places and > to give a description of those valid ip addresses). > > After finding the offending device, was 99% positive it was malicious – > but as I dived into the Rabbit Hole – discovered it was just a stupid > malfunctioning device…a Roku Stick. I’ve also seen this behavior on other > devices that make use of a “Router/IP Sharing” SSID such as “Roku’s Dorm > Mode” or “Internet Sharing” with Windows. > > The Roku generates it’s own SSID “AP Mode” while connecting to our > infrastructure SSID – it’s not bridged – but routed based on the fact that > when you connect your phone or computer to the Roku’s SSID – your assigned > a 192.168.X private IP Address. What I suspect happened in our scenario > (I’ll use your 23.185.0.1 address for example). > > 1. Student Connected Roku to Guest SSID > > 2. Roku Prompted Student to use “Dorm Mode” > > 3. Student Connected to Roku with iPhone or Computer with a “home page” of > our institution’s website. > > 4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 – > Destination: 192.168.X.X – instead of sending it to the “private network” > wifi interface to the user’s iPhone or computer – it sent it out the > “infrastructure network” interface – which based on how a “User” gets into > the table à > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e > – and was assigned the guest unauthenticated policy – denying all traffic – > cept icmps. > > > > I first started suspecting things weren’t as “simple” as they may be when > I noticed Roku’s were “claiming” the IP Addresses of Google – what was > funny was seeing the Controller prevent one Roku from entering the > User-Table with a Google IP Address – *ONLY because another Roku* had > already sourced a packet with Google’s IP Address. > > > > If you add a “any any any deny” with “LOG” option enabled – you can see > ALL the invalid sessions that would have entered the user-table – including > their destinations. > > > I was only able to “partially replicate the behavior” – but it’s still a > strong case. > > A few links down below: > > > How the user gets into the user-table of the controller? - > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e > IP Address Leaking - > https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e > > > Some info from the ArubaOS Hardening Guide > > https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364 > - Page 26 and 27 goes into detail about “validuser” and > “local-valid-users” – “local-valid-users” requires the controller to have > an IP Address on that VLAN interface. There’s also the “Enforce DHCP” > option in each AAA Aruba Profile – essentially a per SSID setting. > > > > > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c > > > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4 > > > > http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/ > -à (BROKEN LINK Now ☹) > Unfortunately the video link I had from commsolutions – they had > presentation demonstrating this issue but it’s a broken link now –one of > their customers for whatever reason had their guests manually enter the ip > addresses onto their ipads – and someone flip-flopped the “IP Address” and > the “Default Gateway”….started denying traffic for the default > gateway….whoops! > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Mike Fitzgerald > *Sent:* Tuesday, September 07, 2021 12:16 PM > *To:* WI
Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
Hi Educause wifi: We use a filter that only allows clients to "have" a valid IP address from "our" range. It' a bit of overhead, but it solves this issue for us. We also say clients listed with addresses that really make no sense. you build a list something like this: netdestination umn-wiredv4-wireless-user-networks network 10.128.0.0 255.224.0.0 network 10.160.0.0 255.240.0.0 network 192.168.157.0 255.255.255.192 network 10.32.253.128 255.255.255.128 network 10.33.9.0 255.255.255.0 description "wiredv4 service ip's for users" add it to valid user: ip access-list session validuser network 127.0.0.0 255.0.0.0 any any deny network 169.254.0.0 255.255.0.0 any any deny network 224.0.0.0 240.0.0.0 any any deny host 255.255.255.255 any any deny network 240.0.0.0 240.0.0.0 any any deny alias umn-wiredv4-wireless-user-networks any any permit any any any deny Something similar is needed for V6. /daniel/ daniel westacott University of Minnesota On Tue, Sep 7, 2021 at 11:04 AM Sidharth Nandury wrote: > So. sigh! > > It seems like an end client either statically or for some unknown reason > got assigned the IP address for these websites. The role that the client > was assigned had a policy to "deny" traffic to the internet (as per > design). The part that we did not know was that when a client is going to a > particular destination, the controllers look at the user table to see if > there is an IP and a route available before even going to the role-based > ACLs. > > Once we blacklisted the client or deleted the client from the user-table, > the websites were accessible again. > > Sid > > On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada > wrote: > >> With 8.6.0.9, no issues. >> >> >> >> (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25 >> >> 35.186.224.25 172.16.122.193 6443 58612 0/0 024 3 >> tunnel 2306 a5 69 11747 17 >> >> 172.16.126.14335.186.224.25 665364 4430/0 024 0 >> tunnel 1718 1a 29 3592 TC 26 >> >> 172.18.91.115 35.186.224.25 656982 4430/0 00 0 >> tunnel 1102 505 14524120 C 29 >> >> 172.16.174.33 35.186.224.25 654373 4430/0 024 0 >> tunnel 2773 6da 9576 1018764TC 21 >> >> 35.186.224.25 172.16.166.198 6443 60052 0/0 024 1 >> tunnel 133 de 371269692 31 >> >> 172.16.172.51 35.186.224.25 663940 4430/0 024 3 >> tunnel 862 5c 17 2849 TC 30 >> >> 172.19.90.133 35.186.224.25 654371 4430/0 024 0 >> tunnel 1509 890 16133426 TC 18 >> >> 172.19.91.45 35.186.224.25 662292 4430/0 024 2 >> tunnel 1630 4d 14 2502 TC 27 >> >> 35.186.224.25 172.16.166.198 6443 60050 0/0 024 14 >> tunnel 133 de 24 8727 31 >> >> 172.16.176.74 35.186.224.25 658973 4430/0 024 2 >> tunnel 1964 236 35 5322 TC 16 >> >> 172.16.176.19335.186.224.25 661015 4430/0 024 1 >> tunnel 2160 10 44 15853 FTC 20 >> >> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs >> *Sent:* Tuesday, September 7, 2021 10:59 AM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from >> wireless network - Aruba >> >> >> >> CAUTION: This email originated from outside of the University. Do not >> click links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> >> Not seeing that issue here. We are on 8.7.1.4 >> >> >> >> (aruba-controller-1) #show datapath session | include 35.186.224.25 >> 35.186.224.25 138.236.104.67 6443 64918 0/0 01 1 >> tunnel 6347 3cc 30750335 15 >> 138.236.82.47 35.186.224.25 657491 4430/0 00 4 >> tunnel 5540 382 179117595 C 30 >> 35.186.224.25 138.236.248.10 6443 54342 0/0 01 1 >> tunnel 972 e20916359 23 >> 35.186.224.25 138.236.82.47 6443 57491 0/0 01
Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
This is very helpful! Thank you. We are planning to implement the validusers-acl like you mentioned and restrict clients to only the IPs that we provide via DHCP. The description is exactly what we are seeing. Christopher, would it be alright if we reached out to you if we have questions? I would hate to re-invent the wheel. Thank you, again. Sid On Tue, Sep 7, 2021 at 3:53 PM Johnson, Christopher wrote: > Sid, > > > We know from personal experience of running into this issue several years > ago. Like David, we’ve instituted a few validuserACLs – (I actually use > aliases for those subnets – so that I can re-use them in other places and > to give a description of those valid ip addresses). > > After finding the offending device, was 99% positive it was malicious – > but as I dived into the Rabbit Hole – discovered it was just a stupid > malfunctioning device…a Roku Stick. I’ve also seen this behavior on other > devices that make use of a “Router/IP Sharing” SSID such as “Roku’s Dorm > Mode” or “Internet Sharing” with Windows. > > The Roku generates it’s own SSID “AP Mode” while connecting to our > infrastructure SSID – it’s not bridged – but routed based on the fact that > when you connect your phone or computer to the Roku’s SSID – your assigned > a 192.168.X private IP Address. What I suspect happened in our scenario > (I’ll use your 23.185.0.1 address for example). > > 1. Student Connected Roku to Guest SSID > > 2. Roku Prompted Student to use “Dorm Mode” > > 3. Student Connected to Roku with iPhone or Computer with a “home page” of > our institution’s website. > > 4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 – > Destination: 192.168.X.X – instead of sending it to the “private network” > wifi interface to the user’s iPhone or computer – it sent it out the > “infrastructure network” interface – which based on how a “User” gets into > the table à > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e > – and was assigned the guest unauthenticated policy – denying all traffic – > cept icmps. > > > > I first started suspecting things weren’t as “simple” as they may be when > I noticed Roku’s were “claiming” the IP Addresses of Google – what was > funny was seeing the Controller prevent one Roku from entering the > User-Table with a Google IP Address – *ONLY because another Roku* had > already sourced a packet with Google’s IP Address. > > > > If you add a “any any any deny” with “LOG” option enabled – you can see > ALL the invalid sessions that would have entered the user-table – including > their destinations. > > > I was only able to “partially replicate the behavior” – but it’s still a > strong case. > > A few links down below: > > > How the user gets into the user-table of the controller? - > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e > IP Address Leaking - > https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e > > > Some info from the ArubaOS Hardening Guide > > https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364 > - Page 26 and 27 goes into detail about “validuser” and > “local-valid-users” – “local-valid-users” requires the controller to have > an IP Address on that VLAN interface. There’s also the “Enforce DHCP” > option in each AAA Aruba Profile – essentially a per SSID setting. > > > > > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c > > > https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4 > > > > http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/ > -à (BROKEN LINK Now ☹) > Unfortunately the video link I had from commsolutions – they had > presentation demonstrating this issue but it’s a broken link now –one of > their customers for whatever reason had their guests manually enter the ip > addresses onto their ipads – and someone flip-flopped the “IP Address” and > the “Default Gateway”….started denying traffic for the default > gateway….whoops! > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Mike Fitzgerald > *Sent:* Tuesday, September 07, 2021 12:16 PM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites > inaccessible from wireless network - Aruba >
RE: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
Sid, We know from personal experience of running into this issue several years ago. Like David, we’ve instituted a few validuserACLs – (I actually use aliases for those subnets – so that I can re-use them in other places and to give a description of those valid ip addresses). After finding the offending device, was 99% positive it was malicious – but as I dived into the Rabbit Hole – discovered it was just a stupid malfunctioning device…a Roku Stick. I’ve also seen this behavior on other devices that make use of a “Router/IP Sharing” SSID such as “Roku’s Dorm Mode” or “Internet Sharing” with Windows. The Roku generates it’s own SSID “AP Mode” while connecting to our infrastructure SSID – it’s not bridged – but routed based on the fact that when you connect your phone or computer to the Roku’s SSID – your assigned a 192.168.X private IP Address. What I suspect happened in our scenario (I’ll use your 23.185.0.1 address for example). 1. Student Connected Roku to Guest SSID 2. Roku Prompted Student to use “Dorm Mode” 3. Student Connected to Roku with iPhone or Computer with a “home page” of our institution’s website. 4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 – Destination: 192.168.X.X – instead of sending it to the “private network” wifi interface to the user’s iPhone or computer – it sent it out the “infrastructure network” interface – which based on how a “User” gets into the table --> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e – and was assigned the guest unauthenticated policy – denying all traffic – cept icmps. I first started suspecting things weren’t as “simple” as they may be when I noticed Roku’s were “claiming” the IP Addresses of Google – what was funny was seeing the Controller prevent one Roku from entering the User-Table with a Google IP Address – ONLY because another Roku had already sourced a packet with Google’s IP Address. If you add a “any any any deny” with “LOG” option enabled – you can see ALL the invalid sessions that would have entered the user-table – including their destinations. I was only able to “partially replicate the behavior” – but it’s still a strong case. A few links down below: How the user gets into the user-table of the controller? - https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e IP Address Leaking - https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e Some info from the ArubaOS Hardening Guide https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364 - Page 26 and 27 goes into detail about “validuser” and “local-valid-users” – “local-valid-users” requires the controller to have an IP Address on that VLAN interface. There’s also the “Enforce DHCP” option in each AAA Aruba Profile – essentially a per SSID setting. https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4 http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/ ---> (BROKEN LINK Now ☹) Unfortunately the video link I had from commsolutions – they had presentation demonstrating this issue but it’s a broken link now –one of their customers for whatever reason had their guests manually enter the ip addresses onto their ipads – and someone flip-flopped the “IP Address” and the “Default Gateway”….started denying traffic for the default gateway….whoops! From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Mike Fitzgerald Sent: Tuesday, September 07, 2021 12:16 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba Some people who received this message don't often get email from fi...@brandeis.edu. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Check your valid user table config to make sure you only allow the IP ranges your DHCP server would give a wireless client. Otherwise, you can end up with user table entries for destination IP's and then those IP's get policed by the controller as you were seeing. Aruba default for that config used to allow any any, which is bad... Mike On Tue, Sep 7, 2021 at 12:04 PM Sidharth Nandury mailto:nandu...@denison.edu>> wrote: So. sigh! It seems like an end client either statically or for some unknown reason got assigned the IP address for these websites. The role that the client was assign
Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
Check your valid user table config to make sure you only allow the IP ranges your DHCP server would give a wireless client. Otherwise, you can end up with user table entries for destination IP's and then those IP's get policed by the controller as you were seeing. Aruba default for that config used to allow any any, which is bad... Mike On Tue, Sep 7, 2021 at 12:04 PM Sidharth Nandury wrote: > So. sigh! > > It seems like an end client either statically or for some unknown reason > got assigned the IP address for these websites. The role that the client > was assigned had a policy to "deny" traffic to the internet (as per > design). The part that we did not know was that when a client is going to a > particular destination, the controllers look at the user table to see if > there is an IP and a route available before even going to the role-based > ACLs. > > Once we blacklisted the client or deleted the client from the user-table, > the websites were accessible again. > > Sid > > On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada > wrote: > >> With 8.6.0.9, no issues. >> >> >> >> (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25 >> >> 35.186.224.25 172.16.122.193 6443 58612 0/0 024 3 >> tunnel 2306 a5 69 11747 17 >> >> 172.16.126.14335.186.224.25 665364 4430/0 024 0 >> tunnel 1718 1a 29 3592 TC 26 >> >> 172.18.91.115 35.186.224.25 656982 4430/0 00 0 >> tunnel 1102 505 14524120 C 29 >> >> 172.16.174.33 35.186.224.25 654373 4430/0 024 0 >> tunnel 2773 6da 9576 1018764TC 21 >> >> 35.186.224.25 172.16.166.198 6443 60052 0/0 024 1 >> tunnel 133 de 371269692 31 >> >> 172.16.172.51 35.186.224.25 663940 4430/0 024 3 >> tunnel 862 5c 17 2849 TC 30 >> >> 172.19.90.133 35.186.224.25 654371 4430/0 024 0 >> tunnel 1509 890 16133426 TC 18 >> >> 172.19.91.45 35.186.224.25 662292 4430/0 024 2 >> tunnel 1630 4d 14 2502 TC 27 >> >> 35.186.224.25 172.16.166.198 6443 60050 0/0 024 14 >> tunnel 133 de 24 8727 31 >> >> 172.16.176.74 35.186.224.25 658973 4430/0 024 2 >> tunnel 1964 236 35 5322 TC 16 >> >> 172.16.176.19335.186.224.25 661015 4430/0 024 1 >> tunnel 2160 10 44 15853 FTC 20 >> >> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs >> *Sent:* Tuesday, September 7, 2021 10:59 AM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from >> wireless network - Aruba >> >> >> >> CAUTION: This email originated from outside of the University. Do not >> click links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> >> Not seeing that issue here. We are on 8.7.1.4 >> >> >> >> (aruba-controller-1) #show datapath session | include 35.186.224.25 >> 35.186.224.25 138.236.104.67 6443 64918 0/0 01 1 >> tunnel 6347 3cc 30750335 15 >> 138.236.82.47 35.186.224.25 657491 4430/0 00 4 >> tunnel 5540 382 179117595 C 30 >> 35.186.224.25 138.236.248.10 6443 54342 0/0 01 1 >> tunnel 972 e20916359 23 >> 35.186.224.25 138.236.82.47 6443 57491 0/0 01 4 >> tunnel 5540 382 18945940 30 >> 138.236.104.6735.186.224.25 664918 4430/0 00 1 >> tunnel 6347 3cd 34538357 C 29 >> 35.186.224.25 138.236.232.120 6443 61505 0/0 01 0 >> tunnel 7052 c15149165 22 >> 138.236.250.8535.186.224.25 654833 4430/0 00 1 >> tunnel 2686 1a 57 16206 C 27 >> 35.186.224.25 138.236.251.120 6443 51735 0/0 01 1 >> tunnel 7060 829 3140 F 13 >> 138.236.250.8535.186.224.25 6
Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
So. sigh! It seems like an end client either statically or for some unknown reason got assigned the IP address for these websites. The role that the client was assigned had a policy to "deny" traffic to the internet (as per design). The part that we did not know was that when a client is going to a particular destination, the controllers look at the user table to see if there is an IP and a route available before even going to the role-based ACLs. Once we blacklisted the client or deleted the client from the user-table, the websites were accessible again. Sid On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada wrote: > With 8.6.0.9, no issues. > > > > (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25 > > 35.186.224.25 172.16.122.193 6443 58612 0/0 024 3 > tunnel 2306 a5 69 11747 17 > > 172.16.126.14335.186.224.25 665364 4430/0 024 0 > tunnel 1718 1a 29 3592 TC 26 > > 172.18.91.115 35.186.224.25 656982 4430/0 00 0 > tunnel 1102 505 14524120 C 29 > > 172.16.174.33 35.186.224.25 654373 4430/0 024 0 > tunnel 2773 6da 9576 1018764TC 21 > > 35.186.224.25 172.16.166.198 6443 60052 0/0 024 1 > tunnel 133 de 371269692 31 > > 172.16.172.51 35.186.224.25 663940 4430/0 024 3 > tunnel 862 5c 17 2849 TC 30 > > 172.19.90.133 35.186.224.25 654371 4430/0 024 0 > tunnel 1509 890 16133426 TC 18 > > 172.19.91.45 35.186.224.25 662292 4430/0 024 2 > tunnel 1630 4d 14 2502 TC 27 > > 35.186.224.25 172.16.166.198 6443 60050 0/0 024 14 > tunnel 133 de 24 8727 31 > > 172.16.176.74 35.186.224.25 658973 4430/0 024 2 > tunnel 1964 236 35 5322 TC 16 > > 172.16.176.19335.186.224.25 661015 4430/0 024 1 > tunnel 2160 10 44 15853 FTC 20 > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs > *Sent:* Tuesday, September 7, 2021 10:59 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from > wireless network - Aruba > > > > CAUTION: This email originated from outside of the University. Do not > click links or open attachments unless you recognize the sender and know > the content is safe. > > > > Not seeing that issue here. We are on 8.7.1.4 > > > > (aruba-controller-1) #show datapath session | include 35.186.224.25 > 35.186.224.25 138.236.104.67 6443 64918 0/0 01 1 > tunnel 6347 3cc 30750335 15 > 138.236.82.47 35.186.224.25 657491 4430/0 00 4 > tunnel 5540 382 179117595 C 30 > 35.186.224.25 138.236.248.10 6443 54342 0/0 01 1 > tunnel 972 e20916359 23 > 35.186.224.25 138.236.82.47 6443 57491 0/0 01 4 > tunnel 5540 382 18945940 30 > 138.236.104.6735.186.224.25 664918 4430/0 00 1 > tunnel 6347 3cd 34538357 C 29 > 35.186.224.25 138.236.232.120 6443 61505 0/0 01 0 > tunnel 7052 c15149165 22 > 138.236.250.8535.186.224.25 654833 4430/0 00 1 > tunnel 2686 1a 57 16206 C 27 > 35.186.224.25 138.236.251.120 6443 51735 0/0 01 1 > tunnel 7060 829 3140 F 13 > 138.236.250.8535.186.224.25 654834 4430/0 00 2 > tunnel 2686 18 152179792 C 27 > > > > --Dan > > > > On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury > wrote: > > Hi All, > > > > Since last Monday we have seen a couple of different websites being > blocked on our Aruba wireless controllers. Spotify has been one of the > sites, as well as all websites hosted on IP 23.185.0.1 (which is our main > institution website - denison.edu). We can confirm that this is being > blocked as we see the "D" (Deny) Flag on the wireless controller. Below is > an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 > controllers seeing this? > > > > (wlc-Thor) #show datapath session | in
RE: [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
With 8.6.0.9, no issues. (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25 35.186.224.25 172.16.122.193 6443 58612 0/0 024 3 tunnel 2306 a5 69 11747 17 172.16.126.14335.186.224.25 665364 4430/0 024 0 tunnel 1718 1a 29 3592 TC 26 172.18.91.115 35.186.224.25 656982 4430/0 00 0 tunnel 1102 505 14524120 C 29 172.16.174.33 35.186.224.25 654373 4430/0 024 0 tunnel 2773 6da 9576 1018764TC 21 35.186.224.25 172.16.166.198 6443 60052 0/0 024 1 tunnel 133 de 371269692 31 172.16.172.51 35.186.224.25 663940 4430/0 024 3 tunnel 862 5c 17 2849 TC 30 172.19.90.133 35.186.224.25 654371 4430/0 024 0 tunnel 1509 890 16133426 TC 18 172.19.91.45 35.186.224.25 662292 4430/0 024 2 tunnel 1630 4d 14 2502 TC 27 35.186.224.25 172.16.166.198 6443 60050 0/0 024 14 tunnel 133 de 24 8727 31 172.16.176.74 35.186.224.25 658973 4430/0 024 2 tunnel 1964 236 35 5322 TC 16 172.16.176.19335.186.224.25 661015 4430/0 024 1 tunnel 2160 10 44 15853 FTC 20 From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Dan Oachs Sent: Tuesday, September 7, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba CAUTION: This email originated from outside of the University. Do not click links or open attachments unless you recognize the sender and know the content is safe. Not seeing that issue here. We are on 8.7.1.4 (aruba-controller-1) #show datapath session | include 35.186.224.25 35.186.224.25 138.236.104.67 6443 64918 0/0 01 1 tunnel 6347 3cc 30750335 15 138.236.82.47 35.186.224.25 657491 4430/0 00 4 tunnel 5540 382 179117595 C 30 35.186.224.25 138.236.248.10 6443 54342 0/0 01 1 tunnel 972 e20916359 23 35.186.224.25 138.236.82.47 6443 57491 0/0 01 4 tunnel 5540 382 18945940 30 138.236.104.6735.186.224.25 664918 4430/0 00 1 tunnel 6347 3cd 34538357 C 29 35.186.224.25 138.236.232.120 6443 61505 0/0 01 0 tunnel 7052 c15149165 22 138.236.250.8535.186.224.25 654833 4430/0 00 1 tunnel 2686 1a 57 16206 C 27 35.186.224.25 138.236.251.120 6443 51735 0/0 01 1 tunnel 7060 829 3140 F 13 138.236.250.8535.186.224.25 654834 4430/0 00 2 tunnel 2686 18 152179792 C 27 --Dan On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury mailto:nandu...@denison.edu>> wrote: Hi All, Since last Monday we have seen a couple of different websites being blocked on our Aruba wireless controllers. Spotify has been one of the sites, as well as all websites hosted on IP 23.185.0.1 (which is our main institution website - denison.edu<http://denison.edu>). We can confirm that this is being blocked as we see the "D" (Deny) Flag on the wireless controller. Below is an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 controllers seeing this? (wlc-Thor) #show datapath session | include 35.186.224.25 Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge PacketsBytes Flags CPU ID - --- - - --- --- --- -- -- --- --- 10.143.203.26 35.186.224.25 652082 4430/0 00 0 tunnel 640 10 0 FDYCA 21 10.143.195.85 35.186.224.25 659767 4430/0 00 0 tunnel 5357 00 0 FDYCA 27 10.143.225.17835.186.224.25 652292 4430/0 00 0 tunnel 6753 10 0 FDYCA 19 10.143.195.85 35.186.224.25 659766 4430/0 00 0 tunnel 5357 10 0 FDYCA 27 (wlc-Thor) #show datapath session | include 23.185.0.1 10.143.228.16 23.185.0.1 659500 4430/0 00 0 tunnel 16789 a0 0 FDYCA 18 10.143.244.15123.185.0.1 658758 443
Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba
Not seeing that issue here. We are on 8.7.1.4 (aruba-controller-1) #show datapath session | include 35.186.224.25 35.186.224.25 138.236.104.67 6443 64918 0/0 01 1 tunnel 6347 3cc 30750335 15 138.236.82.47 35.186.224.25 657491 4430/0 00 4 tunnel 5540 382 179117595 C 30 35.186.224.25 138.236.248.10 6443 54342 0/0 01 1 tunnel 972 e20916359 23 35.186.224.25 138.236.82.47 6443 57491 0/0 01 4 tunnel 5540 382 18945940 30 138.236.104.6735.186.224.25 664918 4430/0 00 1 tunnel 6347 3cd 34538357 C 29 35.186.224.25 138.236.232.120 6443 61505 0/0 01 0 tunnel 7052 c15149165 22 138.236.250.8535.186.224.25 654833 4430/0 00 1 tunnel 2686 1a 57 16206 C 27 35.186.224.25 138.236.251.120 6443 51735 0/0 01 1 tunnel 7060 829 3140 F 13 138.236.250.8535.186.224.25 654834 4430/0 00 2 tunnel 2686 18 152179792 C 27 --Dan On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury wrote: > Hi All, > > Since last Monday we have seen a couple of different websites being > blocked on our Aruba wireless controllers. Spotify has been one of the > sites, as well as all websites hosted on IP 23.185.0.1 (which is our main > institution website - denison.edu). We can confirm that this is being > blocked as we see the "D" (Deny) Flag on the wireless controller. Below is > an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 > controllers seeing this? > > (wlc-Thor) #show datapath session | include 35.186.224.25 > > Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age > Destination TAge PacketsBytes Flags CPU ID > > - --- - - --- --- > --- -- -- --- --- > > 10.143.203.26 35.186.224.25 652082 4430/0 00 0 > tunnel 640 10 0 *FDYCA * 21 > > 10.143.195.85 35.186.224.25 659767 4430/0 00 0 > tunnel 5357 00 0* FDYCA* 27 > > 10.143.225.17835.186.224.25 652292 4430/0 00 0 > tunnel 6753 10 0 * FDYCA * 19 > > 10.143.195.85 35.186.224.25 659766 4430/0 00 0 > tunnel 5357 10 0 *FDYCA * 27 > > > (wlc-Thor) #show datapath session | include 23.185.0.1 > 10.143.228.16 23.185.0.1 659500 4430/0 00 0 > tunnel 16789 a0 0 *FDYCA* 18 > 10.143.244.15123.185.0.1 658758 4430/0 00 0 > tunnel 553 10 0 *FDYCA* 23 > 10.143.228.24723.185.0.1 659063 4430/0 00 0 > tunnel 13188 a6 384*FDYCA* 27 > 10.143.228.24723.185.0.1 659062 4430/0 00 0 > tunnel 13188 a6 384*FDYCA* 27 > 10.143.196.26 23.185.0.1 650851 4430/0 00 0 > tunnel 5631 10 0 *FDYCA* 17 > 10.143.196.26 23.185.0.1 650852 4430/0 00 0 > tunnel 5631 10 0 *FDYCA* 17 > 10.143.196.26 23.185.0.1 650853 4430/0 00 0 > tunnel 5631 10 0 *FDYCA* 17 > > > We have two 7240xm controllers running AOS v8.6.9 in a cluster with a > Mobility Conductor as a VM. We have a ticket open with TAC and have > escalated it up to ERT, but wanted to also reach out to others. > > > Thank you. > > Sid > > > -- > > [image: Denison University] <https://denison.edu> > > *Sidharth S. Nandury* > (He, Him, His) > *Infrastructure and Operations Manager* > Information Technology Services > > 100 West College Street, Granville, OH 43023 <https://deniso.nu/2qF6h7M> | > Burton > Hall <https://denison.edu/map> > Office: 740-587-5533 <1-740-587-5533> | Mobile: 516-314-4413 > <1-516-314-4413> > nand...@denison.edu > https://denison.edu/campus/technology/service-desk > > NOTICE: This email message and all attachments transmitted with it may > contain legally privileged and confidential information intended solely for > the use of the addressee. If the reader of this message is not the intended > recipient, you are hereby notified that any reading, di
Websites inaccessible from wireless network - Aruba
Hi All, Since last Monday we have seen a couple of different websites being blocked on our Aruba wireless controllers. Spotify has been one of the sites, as well as all websites hosted on IP 23.185.0.1 (which is our main institution website - denison.edu). We can confirm that this is being blocked as we see the "D" (Deny) Flag on the wireless controller. Below is an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 controllers seeing this? (wlc-Thor) #show datapath session | include 35.186.224.25 Source IP or MAC Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge PacketsBytes Flags CPU ID - --- - - --- --- --- -- -- --- --- 10.143.203.26 35.186.224.25 652082 4430/0 00 0 tunnel 640 10 0 *FDYCA * 21 10.143.195.85 35.186.224.25 659767 4430/0 00 0 tunnel 5357 00 0* FDYCA* 27 10.143.225.17835.186.224.25 652292 4430/0 00 0 tunnel 6753 10 0 * FDYCA * 19 10.143.195.85 35.186.224.25 659766 4430/0 00 0 tunnel 5357 10 0 *FDYCA * 27 (wlc-Thor) #show datapath session | include 23.185.0.1 10.143.228.16 23.185.0.1 659500 4430/0 00 0 tunnel 16789 a0 0 *FDYCA* 18 10.143.244.15123.185.0.1 658758 4430/0 00 0 tunnel 553 10 0 *FDYCA* 23 10.143.228.24723.185.0.1 659063 4430/0 00 0 tunnel 13188 a6 384*FDYCA* 27 10.143.228.24723.185.0.1 659062 4430/0 00 0 tunnel 13188 a6 384*FDYCA* 27 10.143.196.26 23.185.0.1 650851 4430/0 00 0 tunnel 5631 10 0 *FDYCA* 17 10.143.196.26 23.185.0.1 650852 4430/0 00 0 tunnel 5631 10 0 *FDYCA* 17 10.143.196.26 23.185.0.1 650853 4430/0 00 0 tunnel 5631 10 0 *FDYCA* 17 We have two 7240xm controllers running AOS v8.6.9 in a cluster with a Mobility Conductor as a VM. We have a ticket open with TAC and have escalated it up to ERT, but wanted to also reach out to others. Thank you. Sid -- [image: Denison University] <https://denison.edu> *Sidharth S. Nandury* (He, Him, His) *Infrastructure and Operations Manager* Information Technology Services 100 West College Street, Granville, OH 43023 <https://deniso.nu/2qF6h7M> | Burton Hall <https://denison.edu/map> Office: 740-587-5533 <1-740-587-5533> | Mobile: 516-314-4413 <1-516-314-4413> nand...@denison.edu https://denison.edu/campus/technology/service-desk NOTICE: This email message and all attachments transmitted with it may contain legally privileged and confidential information intended solely for the use of the addressee. If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by phone or by email, and delete this message and all copies and backups thereof. *Please consider the environment before printing this email.* ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Hi Chad, We have an 8 node cluster with roughly 3,500 to 4,000 users on each node. In this case “users” really means IP addresses the way they get counted. So in a dual IPv4/IPv6 stack client environment, a single user may show up as 3 “users” with the IPv4 address, IPv6 link local, and globally routable IPv6 address. Eric Kenny Network Architect | Technology Partner Services Harvard University Information Technology On Sep 3, 2021, at 12:17 PM, Street, Chad A mailto:cstr...@emory.edu>> wrote: For anyone who has applied the suggested fixes and are still having problems -- what are your client loads per controller? I am also very curious on any feedback for anyone who has applied the fixes and has also successfully enabled Airwave ( not central ) without issues reoccurring. Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Patrick McEvilly mailto:patrick_mcevi...@harvard.edu>> Sent: Friday, September 3, 2021 12:03 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) A quick update on where we are at. We had many of the same issues being reported and did the same workarounds. However, we ran into a few further issues that others might want to be aware of. We had one AP group that did not have our AP crash dump server configured and was using the "default" which is the controller itself. ~300 APs in that AP group had core dumped and were trying to write to the controller. There is some mechanism that checks for available space on the controller before dumping. This was jamming up the same queue that is used for other critical processes such as STM. This caused a huge problem for us. We changed this one AP group to point to our dump server and saw immediate relief. All APs came back online and clients reconnected. Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, not 17,000) on our wireless network. We had a cluster event that triggered the controllers to send an invalid IPv4 address to all APs which in turn caused every AP to go into a core dump loop (that was not fun). The only recovery option available was to reboot all controllers. It is believed this is related to 8.7 now supporting IPv6 in the clustering configuration. Our controllers are dual stack but our APs are not. It is possible if the APs were dual stacked they would have used IPv6 and continued to operate over IPv6. Per Aruba removing IPv6 from the controllers is also not an option, we could still experience this problem. This one was a real burn and we continue to be susceptible to this condition. We are anticipating an emergency code release from Aruba in the next few days. Patrick On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Jerry Bucklaew" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of j...@buffalo.edu<mailto:j...@buffalo.edu>> wrote: Scott, I have heard it worked in several places, but we are like you. Put in all the recommended changes last night and it is still happening. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu<mailto:sco...@wcu.edu> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent th
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We have two cluster, one with issues and one without. On the one with issues we are seeing roughly 4500 clients per controller. On the one without issues we are seeing around 2000 clients per controller. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Friday, September 3, 2021 12:17 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For anyone who has applied the suggested fixes and are still having problems -- what are your client loads per controller? I am also very curious on any feedback for anyone who has applied the fixes and has also successfully enabled Airwave ( not central ) without issues reoccurring. Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Patrick McEvilly mailto:patrick_mcevi...@harvard.edu>> Sent: Friday, September 3, 2021 12:03 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) A quick update on where we are at. We had many of the same issues being reported and did the same workarounds. However, we ran into a few further issues that others might want to be aware of. We had one AP group that did not have our AP crash dump server configured and was using the "default" which is the controller itself. ~300 APs in that AP group had core dumped and were trying to write to the controller. There is some mechanism that checks for available space on the controller before dumping. This was jamming up the same queue that is used for other critical processes such as STM. This caused a huge problem for us. We changed this one AP group to point to our dump server and saw immediate relief. All APs came back online and clients reconnected. Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, not 17,000) on our wireless network. We had a cluster event that triggered the controllers to send an invalid IPv4 address to all APs which in turn caused every AP to go into a core dump loop (that was not fun). The only recovery option available was to reboot all controllers. It is believed this is related to 8.7 now supporting IPv6 in the clustering configuration. Our controllers are dual stack but our APs are not. It is possible if the APs were dual stacked they would have used IPv6 and continued to operate over IPv6. Per Aruba removing IPv6 from the controllers is also not an option, we could still experience this problem. This one was a real burn and we continue to be susceptible to this condition. We are anticipating an emergency code release from Aruba in the next few days. Patrick On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Jerry Bucklaew" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU%20on%20behalf%20of%20...@buffalo.edu>> wrote: Scott, I have heard it worked in several places, but we are like you. Put in all the recommended changes last night and it is still happening. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu<mailto:sco...@wcu.edu> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who s
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
For anyone who has applied the suggested fixes and are still having problems -- what are your client loads per controller? I am also very curious on any feedback for anyone who has applied the fixes and has also successfully enabled Airwave ( not central ) without issues reoccurring. Chad chad.str...@emory.edu From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Patrick McEvilly Sent: Friday, September 3, 2021 12:03 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) A quick update on where we are at. We had many of the same issues being reported and did the same workarounds. However, we ran into a few further issues that others might want to be aware of. We had one AP group that did not have our AP crash dump server configured and was using the "default" which is the controller itself. ~300 APs in that AP group had core dumped and were trying to write to the controller. There is some mechanism that checks for available space on the controller before dumping. This was jamming up the same queue that is used for other critical processes such as STM. This caused a huge problem for us. We changed this one AP group to point to our dump server and saw immediate relief. All APs came back online and clients reconnected. Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, not 17,000) on our wireless network. We had a cluster event that triggered the controllers to send an invalid IPv4 address to all APs which in turn caused every AP to go into a core dump loop (that was not fun). The only recovery option available was to reboot all controllers. It is believed this is related to 8.7 now supporting IPv6 in the clustering configuration. Our controllers are dual stack but our APs are not. It is possible if the APs were dual stacked they would have used IPv6 and continued to operate over IPv6. Per Aruba removing IPv6 from the controllers is also not an option, we could still experience this problem. This one was a real burn and we continue to be susceptible to this condition. We are anticipating an emergency code release from Aruba in the next few days. Patrick On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Jerry Bucklaew" wrote: Scott, I have heard it worked in several places, but we are like you. Put in all the recommended changes last night and it is still happening. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccstree2%40EMORY.EDU%7C7010b9354b0144becdd908d96ef458f7%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C637662818017617110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=Py3VDUVjpo1WIQ0y8v3PmIk%2BhoQi4TbaMfP0bDNopCw%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccstree2%40EMORY.EDU%7C7010b9354b01
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
A quick update on where we are at. We had many of the same issues being reported and did the same workarounds. However, we ran into a few further issues that others might want to be aware of. We had one AP group that did not have our AP crash dump server configured and was using the "default" which is the controller itself. ~300 APs in that AP group had core dumped and were trying to write to the controller. There is some mechanism that checks for available space on the controller before dumping. This was jamming up the same queue that is used for other critical processes such as STM. This caused a huge problem for us. We changed this one AP group to point to our dump server and saw immediate relief. All APs came back online and clients reconnected. Two hours later we dropped to ~100 APs connected and only 17 clients (yes 17, not 17,000) on our wireless network. We had a cluster event that triggered the controllers to send an invalid IPv4 address to all APs which in turn caused every AP to go into a core dump loop (that was not fun). The only recovery option available was to reboot all controllers. It is believed this is related to 8.7 now supporting IPv6 in the clustering configuration. Our controllers are dual stack but our APs are not. It is possible if the APs were dual stacked they would have used IPv6 and continued to operate over IPv6. Per Aruba removing IPv6 from the controllers is also not an option, we could still experience this problem. This one was a real burn and we continue to be susceptible to this condition. We are anticipating an emergency code release from Aruba in the next few days. Patrick On 9/3/21, 11:20 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Jerry Bucklaew" wrote: Scott, I have heard it worked in several places, but we are like you. Put in all the recommended changes last night and it is still happening. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cjpb%40buffalo.edu%7Cf58d662303514aa7e05808d96eea39ec%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637662774536265169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=QcSqYr93it4Sn%2Fc1GaKHK4AGNRz77g7qR8SZ4EbPpKA%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Scott, I have heard it worked in several places, but we are like you. Put in all the recommended changes last night and it is still happening. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cjpb%40buffalo.edu%7Cf58d662303514aa7e05808d96eea39ec%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637662774536265169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=QcSqYr93it4Sn%2Fc1GaKHK4AGNRz77g7qR8SZ4EbPpKA%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I don't want to jinx anything but we've been able to keep the issues away for about 10 days now. In addition to the ACL changes we've also enabled the broadcast/multicast optimization, lowered our client rebalancing thresholds, and turn off SNMP for the most part. We're leaning to the client rebalancing being one of the bigger portions of that. Before changing that we had some controllers with 8k clients and some with 10. Wayne Ortman Director, Network Services Office of Information Technology (OIT) 1762 Clifton Road |E154 Office (404) 727-8014 | Cell (470) 312-5754 |wayne.ort...@emory.edu -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Scott Swartzentruber Sent: Friday, September 3, 2021 10:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Cwayne.ortman%40EMORY.EDU%7C770659141b174cfc57d008d96eea3937%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C637662774592961015%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=0yX3iNp6ytwHdhV8JRa8nHQ9SxQj464pArCRAPuoblo%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Has anyone had success in resolving the client drops we have been discussing? We had Aruba on the phone very early this morning - put the recommended ACL in place and did a a rolling restart of all 3 of our controllers. We only had to wait to our first class change to see a huge drop of clients and another at the next class change. Seems clear that the change and the reboot made no difference. back on the phone with Aruba now to see what else they come up with... I don't like this ride anymore - can I get off now? Western Carolina University Scott Swartzentruber Director, Networking and Communications Forsyth Bldg, Room B-16 828.227.3212 sco...@wcu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Sorry I’m late to the party. I got ahold of a great tech at Aruba last night. We finished up with his suggested work-arounds 2-3 hours before the advisory came out. Implemented the client rebalancing threshold change, also at the seemingly popular 15% Turned on ALL the broadcast/multicast optimization switches, at both the VLAN and virtual-ap level. Tweaked the mentioned ACLs (but I doubt that did much. An ACL would just prevent sending the traffic, not wasting cycles generating the traffic.) Tech mentioned SNMP polling could be an issue, but since we’re preferring AMON we left that one alone. Lastly, restarted STM process on each controller. Definitely service impacting but staggering the restart across multiple controllers seemed to help. I hadn’t seen a few of those in the thread. Regards, Colin Colin Randall Manager of Data Networking Information and Technology Solutions (ITS) 1600 Jackson Street, Suite 360 303-384-2208 | crand...@mines.edu From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of "Enfield, Chuck" Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Thursday, September 2, 2021 at 11:06 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) CAUTION: This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. I’d like to suggest sending them home, but if we learned anything last year it’s that home wi-fi isn’t so great either. How many times have you heard, “It works when I’m at home?” Well now we know, not always. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Floyd, Brad Sent: Thursday, September 2, 2021 1:00 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) JD, If we wrap each of them along with their devices in an aluminum foil bubble, each user would have their own collision domain. The MIMO reflections would be awesome, we wouldn’t need more than a single channel architecture, and any channel contention would be self-imposed. Here’s hoping we get to catch up again at the next post-COVID WLPC. Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 11:42 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] “That's been my experience for years. The network works great when there are no students around. My working theory is that students emit RF interference, but research ethics won’t let me run the tests, so we'll never know for sure.” It’s worse than that! They are walking bags of water which absorb the good RF, and their devices transmit the bad RF! It’s a conspiracy I tell ya! We’re going to work with TAC on capturing traffic during a class that is known to have issues. After that, we plan to change the rebalancing threshold as well. Thanks everyone for the feedback! JD From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Enfield, Chuck Date: Thursday, September 2, 2021 at 12:15 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > students returned and we went downhill from there. > > > On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv > on behalf of Rob Harris" robert.har...@culinary.edu> wrote: > >Has anyone seen any details regarding what they consider "Large" > environments? We upgraded during the break, but both before and after > versions are affected. We didn't notice this happening before, should we be > concerned now? > >The "dropped" is 0 and the stm cpu usage is in single digits, but client > count is really low (they come back this weekend as well), could we be in the > clear? > >(asked the SE team and opened a tac call, same questions to them) > >thx > >-Original Message- >From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Jason Healy >Sent: Thursday, September 2, 2021 8:45 AM >To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > >CAUTION: This email originated from outside The Culinary Institute of > America. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > >FWIW, Aruba just posted an advisory regarding this issue: > >Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client > Connectivity Failures in Large Client Environments" > >Good luck to those of you hit by this. My students start coming back this > weekend so I'll be watching this closely! > >Jason >** >Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the message, > copy and paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0 > >** >Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the message, > copy and paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0 > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire community > list. If you want to reply only to the person who sent the message, copy and > paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I’d like to suggest sending them home, but if we learned anything last year it’s that home wi-fi isn’t so great either. How many times have you heard, “It works when I’m at home?” Well now we know, not always. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Floyd, Brad Sent: Thursday, September 2, 2021 1:00 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) JD, If we wrap each of them along with their devices in an aluminum foil bubble, each user would have their own collision domain. The MIMO reflections would be awesome, we wouldn’t need more than a single channel architecture, and any channel contention would be self-imposed. Here’s hoping we get to catch up again at the next post-COVID WLPC. Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 11:42 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] “That's been my experience for years. The network works great when there are no students around. My working theory is that students emit RF interference, but research ethics won’t let me run the tests, so we'll never know for sure.” It’s worse than that! They are walking bags of water which absorb the good RF, and their devices transmit the bad RF! It’s a conspiracy I tell ya! We’re going to work with TAC on capturing traffic during a class that is known to have issues. After that, we plan to change the rebalancing threshold as well. Thanks everyone for the feedback! JD From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Enfield, Chuck mailto:cae...@psu.edu>> Date: Thursday, September 2, 2021 at 12:15 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" mailto:cstr...@emory.edu>> Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Rob Harris mailto:robert.har...@culinary.edu>> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
JD, If we wrap each of them along with their devices in an aluminum foil bubble, each user would have their own collision domain. The MIMO reflections would be awesome, we wouldn’t need more than a single channel architecture, and any channel contention would be self-imposed. Here’s hoping we get to catch up again at the next post-COVID WLPC. Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 11:42 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] “That's been my experience for years. The network works great when there are no students around. My working theory is that students emit RF interference, but research ethics won’t let me run the tests, so we'll never know for sure.” It’s worse than that! They are walking bags of water which absorb the good RF, and their devices transmit the bad RF! It’s a conspiracy I tell ya! We’re going to work with TAC on capturing traffic during a class that is known to have issues. After that, we plan to change the rebalancing threshold as well. Thanks everyone for the feedback! JD From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Enfield, Chuck mailto:cae...@psu.edu>> Date: Thursday, September 2, 2021 at 12:15 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" mailto:cstr...@emory.edu>> Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Rob Harris mailto:robert.har...@culinary.edu>> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in t
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
“That's been my experience for years. The network works great when there are no students around. My working theory is that students emit RF interference, but research ethics won’t let me run the tests, so we'll never know for sure.” It’s worse than that! They are walking bags of water which absorb the good RF, and their devices transmit the bad RF! It’s a conspiracy I tell ya! We’re going to work with TAC on capturing traffic during a class that is known to have issues. After that, we plan to change the rebalancing threshold as well. Thanks everyone for the feedback! JD From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Enfield, Chuck Date: Thursday, September 2, 2021 at 12:15 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" mailto:cstr...@emory.edu>> Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Rob Harris mailto:robert.har...@culinary.edu>> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FF7.DF429460] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" mailto:cstr...@emory.edu>> Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Rob Harris mailto:robert.har...@culinary.edu>> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FF4.0F7F8150] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We are on 8.7.1.4 with 6,300 WAPs and 6 controllers. We had to upgrade to 8.7 because of the newer model of WAPs ___ Rita Barrantes, PhD, PMP Director, Technology Services and Support Networks and Telecomm | IT Assessments 832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu> [cid:image001.png@01D79FE7.35A3D4B0] From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Joe Walker Sent: Thursday, September 2, 2021 10:16 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We currently aren't experiencing this issue but these symptoms (stm crashes, controller crashes) were identical to the issues we saw in early 2019 (8.5.0.1) An aggressive rebalance of users/ap's (and eventually an upgrade to 8.5.0.3) fixed this for us. I mention this because we are currently sitting on 8.5.0.13 and don't seem to be having the issues reported by others in the same code so I wonder if there is credence to the load balance thresholds playing a part. Campus cluster 2x7240's 1X7240XM 22K connected devices Thanks, Joe Joe Walker Network and Telecommunication Services Kansas State University (785)532-4997 f...@ksu.edu<mailto:f...@ksu.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Smith, Nayef mailto:nayef.z.sm...@emory.edu>> Sent: Thursday, September 2, 2021 9:19 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This email originated from outside of K-State. "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image002.png@01D79FE7.35A3D4B0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But y
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Nayef – that’s what we did. Down to 15% ___ Rita Barrantes, PhD, PMP Director, Technology Services and Support Networks and Telecomm | IT Assessments 832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu> [cid:image002.png@01D79FE6.EC8BFA50] From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:13 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We were seeing one of our cluster members taking on ~7000+ active clients while two others were in the low double digits. Our previous threshold was at 50%, which seems to have previously been the default. Once we adjusted our thresholds down to 20%, we saw improvements. We then went down to 15%. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image003.png@01D79FE6.EC8BFA50] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Rob Harris mailto:robert.har...@culinary.edu>> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image003.png@01D79FE6.EC8BFA50] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<ma
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Any chance you could be experiencing the issues outlined in today's advisory? https://mail.google.com/mail/u/0?ui=2=b16ec71188=0.1=msg-f:1709803220174325554=17ba70f8464f4732=att=inline On Thu, Sep 2, 2021 at 10:20 AM Enfield, Chuck wrote: > Between 5k and 6k clients on a 7240xm is where we started seeing problems. > Lighter loaded controllers were OK. > -- > *From:* "Street, Chad A" > *Sent:* Thursday, September 2, 2021 11:03 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > > We are a balanced cluster, notes about load below: > > > "I’m also noticing that there are much fewer clients on this controller, > and that ratio doesn’t seem to be improving." > > To this point, the action we took that seemed to help the most was > adjusting our active client load balancing threshold. We dropped it > significantly to force clients to balance across controllers. Once we got > below ~5000 active clients per controller, we stopped seeing the mass > client connection issues. > > We still have a controller that hasn't taken significant load, but now > that we've been running without major issues for the past few days, we're > reluctant to touch the setting again. > -- > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Rob Harris < > robert.har...@culinary.edu> > *Sent:* Thursday, September 2, 2021 10:59 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> > *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > > > For those of you who have experienced this, what was your user load and > how were your clusters operating (balancing, active/standby) ? > > > > I wonder if there’s a threshold.. > > > > Thx! > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Smith, Nayef > *Sent:* Thursday, September 2, 2021 10:20 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > > > > > > > > "I’m also noticing that there are much fewer clients on this controller, > and that ratio doesn’t seem to be improving." > > > > To this point, the action we took that seemed to help the most was > adjusting our active client load balancing threshold. We dropped it > significantly to force clients to balance across controllers. Once we got > below ~5000 active clients per controller, we stopped seeing the mass > client connection issues. > > > > We still have a controller that hasn't taken significant load, but now > that we've been running without major issues for the past few days, we're > reluctant to touch the setting again. > > > > > > Nayef Z. Smith | *Network Services* | Voice: 404-727-6019 > > > > > > -- > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Davis, Jonathan Alan < > jonath...@unc.edu> > *Sent:* Thursday, September 2, 2021 9:27 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> > *Subject:* Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > > > > Lee, don’t you bring your bad Cisco-juju to this conversation! :-) > > > > Now that Lee has been properly handled, this is probably a great > opportunity to say ‘hello’ to the greater list. > > > > Hello! > > > > Last night, we (UNC) restarted the controller used to test the firewall > policy. Despite Aruba’s advisory, we’ve been led to believe that restarting > STM may not be enough, and restarting the whole controller may be required > to resolve high STM CPU utilization. > > > > This morning we are keeping a close eye on that controller. While STM is > surging well past 100%, it seems to be averaging much closer to 95%. > > > > However… > > We also only have about 7,000 users connected across the cluster. It will > be interesting to see what happens as the day progresses and students wak
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Rob Harris Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FE9.A97E6BF0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large cla
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We currently aren't experiencing this issue but these symptoms (stm crashes, controller crashes) were identical to the issues we saw in early 2019 (8.5.0.1) An aggressive rebalance of users/ap's (and eventually an upgrade to 8.5.0.3) fixed this for us. I mention this because we are currently sitting on 8.5.0.13 and don't seem to be having the issues reported by others in the same code so I wonder if there is credence to the load balance thresholds playing a part. Campus cluster 2x7240's 1X7240XM 22K connected devices Thanks, Joe Joe Walker Network and Telecommunication Services Kansas State University (785)532-4997 f...@ksu.edu<mailto:f...@ksu.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Smith, Nayef Sent: Thursday, September 2, 2021 9:19 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This email originated from outside of K-State. "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:dd82bb8e-7a57-4a56-8047-d94ceebd7bec] From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Lee H Badman Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > stu
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We were seeing one of our cluster members taking on ~7000+ active clients while two others were in the low double digits. Our previous threshold was at 50%, which seems to have previously been the default. Once we adjusted our thresholds down to 20%, we saw improvements. We then went down to 15%. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:d4681667-5ec0-49ec-b547-ee2c893c1e10] From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Rob Harris Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FE9.A97E6BF0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck > mailto:cae...@psu.edu>> wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We are experiencing the same issues. We rebooted this morning. ___ Rita Barrantes, PhD, PMP Director, Technology Services and Support Networks and Telecomm | IT Assessments 832-842-4702 | rbarran...@uh.edu<mailto:rbarran...@uh.edu> [cid:image001.png@01D79FE2.62D01BB0] From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 9:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image002.png@01D79FE2.62D01BB0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck > mailto:cae...@psu.edu>> wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Rob Harris Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FE9.A97E6BF0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:image001.png@01D79FE9.A97E6BF0] From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Davis, Jonathan Alan mailto:jonath...@unc.edu>> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Lee H Badman mailto:lhbad...@syr.edu>> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck > mailto:cae...@psu.edu>> wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > students returned and we went downhill from there. > > >
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
"I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 [cid:dd82bb8e-7a57-4a56-8047-d94ceebd7bec] From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Lee H Badman Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > students returned and we went downhill from there. > > > On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv > on behalf of Rob Harris" robert.har...@culinary.edu> wrote: > >Has anyone seen any details regarding what they consider "Large" > environments? We upgraded during the break, but both before and after > versions are affected. We didn't notice this happening before, should we be > concerned now? > >The "dropped" is 0 and the stm cpu usage is in single digits, but client > count is really low (they come back this weekend as well), could we be in the > clear? > >(asked the SE team and opened a tac call, same questions to them) > >thx > >-Original Message- >From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Jason Healy >Sent: Thursday, September 2, 2021 8:45 AM >To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else >
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu<mailto:jonath...@unc.edu> +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Lee H Badman Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -Original Message- > From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > students returned and we went downhill from there. > > > On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv > on behalf of Rob Harris" robert.har...@culinary.edu> wrote: > >Has anyone seen any details regarding what they consider "Large" > environments? We upgraded during the break, but both before and after > versions are affected. We didn't notice this happening before, should we be > concerned now? > >The "dropped" is 0 and the stm cpu usage is in single digits, but client > count is really low (they come back this weekend as well), could we be in the > clear? > >(asked the SE team and opened a tac call, same questions to them) > >thx > >-Original Message- >From: The EDUCAUSE Wireless Issues Community Group Listserv > On Behalf Of Jason Healy >Sent: Thursday, September 2, 2021 8:45 AM >To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > >CAUTION: This email originated from outside The Culinary Institute of > America. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > >FWIW, Aruba just posted an advisory regarding this issue: > >Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client > Connectivity Failures in Large Client Environments" > >Good luck to those of you hit by this. My students start coming back this > weekend so I'll be watching this closely! > >Jason >** >Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the message, > copy and paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We see about 28K devices peak and so far haven't seen the issue crop up. We did just upgrade to 8.7.1.4 just before the Semester begun, because we were getting smacked by the false radar detection bug that was causing AP reboots. On 9/2/21 8:50 AM, Rob Harris wrote: Has anyone seen any details regarding what they consider "Large" environments? We upgraded during the break, but both before and after versions are affected. We didn't notice this happening before, should we be concerned now? The "dropped" is 0 and the stm cpu usage is in single digits, but client count is really low (they come back this weekend as well), could we be in the clear? (asked the SE team and opened a tac call, same questions to them) thx -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Jason Healy Sent: Thursday, September 2, 2021 8:45 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) CAUTION: This email originated from outside The Culinary Institute of America. Do not click links or open attachments unless you recognize the sender and know the content is safe. FWIW, Aruba just posted an advisory regarding this issue: Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity Failures in Large Client Environments" Good luck to those of you hit by this. My students start coming back this weekend so I'll be watching this closely! Jason ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community -- Mike Davis IT - University of Delaware - 302.831.8756 Newark, DE 19716 Email da...@udel.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
When we had the problem last year, we couldn't see the high CPU usage much of the time. The best way to tell if it's crashing is to look at the service uptime. If it's been up for days or weeks you probably don't have the problem. Hours, then you probably do. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Rob Harris Sent: Thursday, September 2, 2021 8:50 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Has anyone seen any details regarding what they consider "Large" environments? We upgraded during the break, but both before and after versions are affected. We didn't notice this happening before, should we be concerned now? The "dropped" is 0 and the stm cpu usage is in single digits, but client count is really low (they come back this weekend as well), could we be in the clear? (asked the SE team and opened a tac call, same questions to them) thx -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Jason Healy Sent: Thursday, September 2, 2021 8:45 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) CAUTION: This email originated from outside The Culinary Institute of America. Do not click links or open attachments unless you recognize the sender and know the content is safe. FWIW, Aruba just posted an advisory regarding this issue: Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity Failures in Large Client Environments" Good luck to those of you hit by this. My students start coming back this weekend so I'll be watching this closely! Jason ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7Ce4da0822a6a143b7424d08d96e104391%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661838403488619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=L8kDlM2Vy7try2q1QdRgBCpOJKQiKGDTkUyY8%2FevgLU%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7Ce4da0822a6a143b7424d08d96e104391%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661838403488619%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=L8kDlM2Vy7try2q1QdRgBCpOJKQiKGDTkUyY8%2FevgLU%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
That's been my experience for years. The network works great when there are no
students around. My working theory is that students emit RF interference, but
research ethics won’t let me run the tests, so we'll never know for sure.
-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv
On Behalf Of Patrick McEvilly
Sent: Thursday, September 2, 2021 8:56 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing
any issues in the fall with large classrooms and delayed connection times
(Aruba 8.5.0.13)
Speaking from experience, I would be very concerned. We had no issues until
students returned and we went downhill from there.
On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on
behalf of Rob Harris" wrote:
Has anyone seen any details regarding what they consider "Large"
environments? We upgraded during the break, but both before and after versions
are affected. We didn't notice this happening before, should we be concerned
now?
The "dropped" is 0 and the stm cpu usage is in single digits, but client
count is really low (they come back this weekend as well), could we be in the
clear?
(asked the SE team and opened a tac call, same questions to them)
thx
-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv
On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
seeing any issues in the fall with large classrooms and delayed connection
times (Aruba 8.5.0.13)
CAUTION: This email originated from outside The Culinary Institute of
America. Do not click links or open attachments unless you recognize the sender
and know the content is safe.
FWIW, Aruba just posted an advisory regarding this issue:
Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity
Failures in Large Client Environments"
Good luck to those of you hit by this. My students start coming back this
weekend so I'll be watching this closely!
Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3Dreserved=0
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Speaking from experience, I would be very concerned. We had no issues until
students returned and we went downhill from there.
On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on
behalf of Rob Harris" wrote:
Has anyone seen any details regarding what they consider "Large"
environments? We upgraded during the break, but both before and after versions
are affected. We didn't notice this happening before, should we be concerned
now?
The "dropped" is 0 and the stm cpu usage is in single digits, but client
count is really low (they come back this weekend as well), could we be in the
clear?
(asked the SE team and opened a tac call, same questions to them)
thx
-Original Message-
From: The EDUCAUSE Wireless Issues Community Group Listserv
On Behalf Of Jason Healy
Sent: Thursday, September 2, 2021 8:45 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else
seeing any issues in the fall with large classrooms and delayed connection
times (Aruba 8.5.0.13)
CAUTION: This email originated from outside The Culinary Institute of
America. Do not click links or open attachments unless you recognize the sender
and know the content is safe.
FWIW, Aruba just posted an advisory regarding this issue:
Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity
Failures in Large Client Environments"
Good luck to those of you hit by this. My students start coming back this
weekend so I'll be watching this closely!
Jason
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at https://www.educause.edu/community
**
Replies to EDUCAUSE Community Group emails are sent to the entire community
list. If you want to reply only to the person who sent the message, copy and
paste their email address and forward the email reply. Additional participation
and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Has anyone seen any details regarding what they consider "Large" environments? We upgraded during the break, but both before and after versions are affected. We didn't notice this happening before, should we be concerned now? The "dropped" is 0 and the stm cpu usage is in single digits, but client count is really low (they come back this weekend as well), could we be in the clear? (asked the SE team and opened a tac call, same questions to them) thx -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Jason Healy Sent: Thursday, September 2, 2021 8:45 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) CAUTION: This email originated from outside The Culinary Institute of America. Do not click links or open attachments unless you recognize the sender and know the content is safe. FWIW, Aruba just posted an advisory regarding this issue: Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity Failures in Large Client Environments" Good luck to those of you hit by this. My students start coming back this weekend so I'll be watching this closely! Jason ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7C9624098759e143958e1708d96e0f8742%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637661835282336222%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=%2Bram23JxP8FS6%2BUruT13pfiX%2F5z8mYsT5yywvQeWTTo%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
FWIW, Aruba just posted an advisory regarding this issue: Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client Connectivity Failures in Large Client Environments" Good luck to those of you hit by this. My students start coming back this weekend so I'll be watching this closely! Jason ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Just a quick update that might be good information for others to have. We were running 8.7.1.3 since June 10th with no issues until yesterday. We were seeing STM crash across several controllers and caused a significant outage in our wireless service. We did the same things others here were told to do, remove SNMP, amp, add the FW rule etc. We did an emergency upgrade to 8.7.1.4 today to address a bug that caused STM to crash. We ran into another issue after the upgrade to 8.7.1.4 that would not allow about 50% of our APs to rejoin the controllers. It is unknown if the issue was introduced in the new code or if it was pre-existing (unlikely pre-existing as we had to do a full controller reload last night and all the APs came back just fine). Over the course of the last 4-5 hours the remaining APs are still trickling in but we are not at 100%. TAC are analyzing everything they captured on our marathon Zoom session. For now, we are in a holding pattern and waiting for direction on how to proceed. If we make any breakthroughs we will be sure to share it here. Patrick From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of "Enfield, Chuck" Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Wednesday, September 1, 2021 at 5:29 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We feel your pain, Patrick! Keep up the good fight. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Patrick McEvilly Sent: Wednesday, September 1, 2021 5:25 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will hold off on providing details for now but when you have to push a code upgrade in the middle of the day on the first day class it’s been a rough day. We hit some major issues related to STM and then other fall out after doing the required code upgrade. We pushed the changes below at 2am this morning. It did help a bit, but issues resurfaced again at 10am. We are still on a call with Aruba TAC and don’t have anything at this time to share that would help others. Patrick From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, September 1, 2021 at 5:13 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) To all chiming in regarding the Aruba issues – thank you! I love seeing the collaboration and detail sharing. Chad – will be curious to hear if you push the band-aids to production and re-enable airwaves, if this helps your situation. -Cody From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 3:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> __
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We had disabled "Prohibit ARP Spoofing" at one point to appease the 14+ code. The issue was resolved in a later release and we enabled it again. We are not currently seeing any issues 8.5.0.11, moving to 13 now. Thanks From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Enfield, Chuck Sent: Wednesday, September 1, 2021 4:29 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) EXTERNAL EMAIL We feel your pain, Patrick! Keep up the good fight. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Patrick McEvilly Sent: Wednesday, September 1, 2021 5:25 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will hold off on providing details for now but when you have to push a code upgrade in the middle of the day on the first day class it's been a rough day. We hit some major issues related to STM and then other fall out after doing the required code upgrade. We pushed the changes below at 2am this morning. It did help a bit, but issues resurfaced again at 10am. We are still on a call with Aruba TAC and don't have anything at this time to share that would help others. Patrick From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, September 1, 2021 at 5:13 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) To all chiming in regarding the Aruba issues - thank you! I love seeing the collaboration and detail sharing. Chad - will be curious to hear if you push the band-aids to production and re-enable airwaves, if this helps your situation. -Cody From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 3:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We feel your pain, Patrick! Keep up the good fight. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Patrick McEvilly Sent: Wednesday, September 1, 2021 5:25 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will hold off on providing details for now but when you have to push a code upgrade in the middle of the day on the first day class it's been a rough day. We hit some major issues related to STM and then other fall out after doing the required code upgrade. We pushed the changes below at 2am this morning. It did help a bit, but issues resurfaced again at 10am. We are still on a call with Aruba TAC and don't have anything at this time to share that would help others. Patrick From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, September 1, 2021 at 5:13 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) To all chiming in regarding the Aruba issues - thank you! I love seeing the collaboration and detail sharing. Chad - will be curious to hear if you push the band-aids to production and re-enable airwaves, if this helps your situation. -Cody From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 3:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F56.E9F8F5D0] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.ED
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Don't forgot the nasty Lenovo vantage software T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Turner, Ryan H Sent: Wednesday, September 1, 2021 5:12 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Glad I brought this up. Is it possible that Cisco environments have evaded this? Seems as though the ARP flooding via iOS 14 would be something that would menace all the manufacturers. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 5:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F54.94BB2180] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much.
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I will hold off on providing details for now but when you have to push a code upgrade in the middle of the day on the first day class it’s been a rough day. We hit some major issues related to STM and then other fall out after doing the required code upgrade. We pushed the changes below at 2am this morning. It did help a bit, but issues resurfaced again at 10am. We are still on a call with Aruba TAC and don’t have anything at this time to share that would help others. Patrick From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Cody Ensanian Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Wednesday, September 1, 2021 at 5:13 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) To all chiming in regarding the Aruba issues – thank you! I love seeing the collaboration and detail sharing. Chad – will be curious to hear if you push the band-aids to production and re-enable airwaves, if this helps your situation. -Cody From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 3:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F56.585D43B0] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 m
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I'm speculating a bit, but Aruba does a lot of stuff with ARP if features like bcast ARP suppression, convert bacst to unicast, and BC/MC optimization are enabled. I assume Cisco has some similar features, but perhaps not all of them? Or maybe one key feature is causing most of the trouble for Aruba. I also know that some of the ARP processing Aruba does on the controllers helps reduce the amount of ARP that reaches the underpinning network. I'm sure many of us have ARP policers kicking in right now. I hate to think about what our switches and routers would be struggling with if the controllers didn't manage this stuff like they do. We may have to pick our poison. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 5:13 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Glad I brought this up. Is it possible that Cisco environments have evaded this? Seems as though the ARP flooding via iOS 14 would be something that would menace all the manufacturers. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 5:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F55.B5EE9F70] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come i
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Ryan, Do you have multicast enabled ?What is the mandatory rate you are using in the classrooms? We just had some issues with this not on Aruba . Bryan Tolka Sent from my iPhone On Sep 1, 2021, at 5:00 PM, Street, Chad A wrote: Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F14.D04A94F0] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Cbtolka%40HSC.WVU.EDU%7Cdac0d4e893364f64b8a108d96d8b964f%7Ca2d1f95f851044248ae15c596bdbd578%7C0%7C0%7C637661268570757361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=tYz0OWkhXLFkiLNk3U2PeYLMMbT%2B8dAiUTa1gRmKaIg%3D=0> ** Replies to EDUCAUSE Community Group ema
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
To all chiming in regarding the Aruba issues - thank you! I love seeing the collaboration and detail sharing. Chad - will be curious to hear if you push the band-aids to production and re-enable airwaves, if this helps your situation. -Cody From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 3:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F43.704FEB70] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire communi
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Glad I brought this up. Is it possible that Cisco environments have evaded this? Seems as though the ARP flooding via iOS 14 would be something that would menace all the manufacturers. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Street, Chad A Sent: Wednesday, September 1, 2021 5:01 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu<mailto:chad.str...@emory.edu> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Cody Ensanian mailto:censa...@uccs.edu>> Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F54.94BB2180] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent th
Re: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Cody and all... We are also seeing STM spikes that are impacting associations. We have also disabled all our polling ( Airwave, Orion, etc ) and reduced the client load balancing thresholds so that we have around 4K clients per controller. This seemed to help a great deal. After working with Aruba today, my understanding of the primary cause of the STM spikes is due to the MM polling the MCs. With large client loads on the MCs ( combined with all the other SNMP polling going on ), this seems to take longer and sometimes does not work. When it does not work, it bootstraps which spikes the STM process. The suggested band-aid is to block the GUI polling traffic between the MM and MC. You will lose the GUI information from your MM, but all the MC information is still present. We have applied this to our lab and we are going to push to production tonight to see if it helps. If it does help, we plan on turning back up our monitoring tools ( Airwave ). fingers crossed here is how to block the traffic: cd /md/yourrootlocation firewall-cp ipv4 deny any proto 6 ports 15260 15261 position 1 ! Chad chad.str...@emory.edu From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F14.D04A94F0] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccstree2%40EMORY.EDU%7C5708a9f9983c4d7c2e2a08d96d5eeb0d%7Ce004fb9cb0a4424fbcd0322606d5df38%7C0%7C0%7C637661076706815129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=l9j1wMMeqkODSIIY%2BItYoCpXWA3zx8poT1eW9r7q74I%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be fou
Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
This happened to us on the first day of classes. It's not your RADIUS. Some problem where, under load, users cannot associate to the AP. The STM process is overcommitted and can't respond appropriately. Aruba advised us to shutdown openflow and the next day the problem was gone. The next steps they advised were to minimize syslog and SNMP. They have all our log dumps and are analyzing. I believe U of Tenn has the problem too but they are not able to shutdown openflow and so are still experiencing the problem. I was not involved in the nitty-gritty so that's about as much as I can tell you. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Turner, Ryan H Sent: Wednesday, September 1, 2021 11:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://urldefense.com/v3/__https://www.educause.edu/community__;!!PhOWcWs!ho0BhxNlSHkWW0J7YEUqzngqoDwe9GIPHFihKuvY50L85BHRuBvxtX4xjPiUWxabFtA$> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Please see my 12:05 response if you missed it. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Steve Smith Sent: Wednesday, September 1, 2021 12:00 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I wouldn't mind seeing the arp limiting client filter as well. Thank you, Steve Steve Smith Network Administrator II Network and Telecommunications Services Aims Community College 970.339.6565 On Wed, Sep 1, 2021 at 9:57 AM Laramie Combs mailto:comb...@appstate.edu>> wrote: HEy Chuck - would you mind sharing that arp limiting client filter with me? We are seeing some new traffic patterns where it looks like user devices are just walking their subnets, and arping for everything -Laramie On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck mailto:cae...@psu.edu>> wrote: We've seen the CPU problem, but I don't think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F2A.AE293D20] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__nam10.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fwww.educause.edu-252Fcommunity-26data-3D04-257C01-257Ccae104-2540PSU.EDU-257C23c7b4692be5427984b208d96d5eeb66-257C7cf48d453ddb4389a9c1c115
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I wouldn't mind seeing the arp limiting client filter as well. Thank you, Steve Steve Smith Network Administrator II Network and Telecommunications Services Aims Community College 970.339.6565 On Wed, Sep 1, 2021 at 9:57 AM Laramie Combs wrote: > HEy Chuck - would you mind sharing that arp limiting client filter with me? > > We are seeing some new traffic patterns where it looks like user devices > are just walking their subnets, and arping for everything > > -Laramie > > On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck wrote: > >> We’ve seen the CPU problem, but I don’t think it resulted in Auth >> problems here. It may have and we just missed it because the more severe >> problems it caused masked them. >> >> >> >> BTW, in our case reducing the amount of ARP calmed the CPU. We applied a >> filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless >> clients could send and it smoothed out the spikes. >> >> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Cody Ensanian >> *Sent:* Wednesday, September 1, 2021 11:41 AM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall >> with large classrooms and delayed connection times (Aruba 8.5.0.13) >> >> >> >> I’m hearing issues of high cpu utilization for STM on the controllers >> causing issues. Maybe check your controllers and see if you are seeing the >> high cpu use for STM. Heard earlier today from our SE that Aruba has >> “identified the issue and is working on a fix.” I suggest opening the TAC >> case so they can track it better, and help them hone in on a fix better. >> We’re seeing the high cpu use on one of our controllers (but this >> controller also has higher client load). However, we have not had a flood >> of calls to our help desk for wireless issues (not saying they aren’t >> happening). Our SE also said if you’re experiencing the issue, disabling >> any system or process level debugging as helped, as well as disabling any >> SNMP polling. >> >> >> >> -Cody >> >> UCCS >> >> >> >> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Turner, Ryan H >> *Sent:* Wednesday, September 1, 2021 9:27 AM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* [WIRELESS-LAN] Anyone else seeing any issues in the fall with >> large classrooms and delayed connection times (Aruba 8.5.0.13) >> >> >> >> This is a stab in the dark. With the University mostly shutdown since >> the Spring of 2020 (=not operating in standard mode and most people work >> from home), we got campus upgraded from 6.X to 8.X code base. We’ve also >> installed many 515 series APs. We are getting a large number of complaints >> in large classrooms that connecting to things like eduroam takes a long >> time. Looking into the connection, we see many incomplete RADIUS >> challenges. The general complaints are ‘we come into the classroom, and >> for some folks it can take up to 5 minutes to get connected’. The odd >> thing is that our RADIUS infrastructure is very large, polished and load >> shared, and we can see no performance issues with any of the RADIUS >> servers. We have begun reducing power in the large classrooms to make >> association issues better, but so far that hasn’t changed much. We >> anticipate opening a ticket with Aruba, soon. We do seem to see the most >> complaints in the big classrooms. But I do keep going back to the RADIUS >> Challenges incomplete. I know if no reason for those not to complete >> unless the connection is broken midway. >> >> >> >> Has anyone else seen something like this? >> >> >> >> Ryan Turner >> >> Head of Networking >> >> Communication Technologies | Information Technology Services >> >> r...@unc.edu >> >> +1 919 445 0113 (Office) >> >> +1 919 274 7926 (Mobile) >> >> >> >> ** >> Replies to EDUCAUSE Community Group emails are sent to the entire >> community list. If you want to reply only to the person who sent the >> message, copy and paste their email address and forward the email reply. >> Additional participation and subscription information can be found at >> https://www.educause.edu/community >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__nam10.safelinks.protection.outlook.com_-3Fu
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
In our situation, we actually measure the packets over the wire to judge RADIUS response. I know precisely when I get a RADIUS timeout and what the average RTT as well as average response time for MAC and 802.1X authentications. So I believe out environment is clean. With that said, I am going to get for the timeouts on the controllers. Thank you. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Enfield, Chuck Sent: Wednesday, September 1, 2021 11:42 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We're not having any unusual problems now, but we have in the past. Two suggestions I can offer are: * Search your controller syslog for "Authentication server request timeout". This will tell you if the controllers are sending auth requests and not getting replies back. We've had this happen when RDAIUS servers report being fat and happy. Best explanation I can offer is that VMs sometimes lie. * Check the controller 802.1X counters to make sure they're not throttling authentications. https://community.arubanetworks.com/blogs/ssasi1/2020/10/28/how-does-auth-throttling-feature-work-and-what-are-the-associated-cli-commands. If this does occur, it tends to happen at times of high user mobility. Good luck. Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 11:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7Cbf80edde3d5a412daf6108d96d5cfb72%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661068387237277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=HBe5V3IJtEi%2FhbE4qSWswW3PQepfmQCI1Hn1Q%2Fi%2BCm8%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Hey Laramine/Chuck, The ARP issue most likely the Lenovo Vantage software or IOS 14. Another option outside of filtering is to enable prohibit ip spoofing and arp spoofing. T.J. Norton Wireless Network Architect Network Operations (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Laramie Combs Sent: Wednesday, September 1, 2021 11:57 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) HEy Chuck - would you mind sharing that arp limiting client filter with me? We are seeing some new traffic patterns where it looks like user devices are just walking their subnets, and arping for everything -Laramie On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck mailto:cae...@psu.edu>> wrote: We’ve seen the CPU problem, but I don’t think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:17ba213cca04cff311] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C7bfb23fbcabf4dbaa9eb08d96d614285%7Cbaf8218eb3024465a9934a39c97251b2
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Here's what we did. We had a pcap that suggested only about 1% of clients would be affected by this filter, but it cut our ARP almost in half. We made the change last spring in our res halls which were almost fully occupied, and we've not traced user complaints back to this yet. [cid:image001.png@01D79F29.7E132B60] From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Laramie Combs Sent: Wednesday, September 1, 2021 11:58 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) HEy Chuck - would you mind sharing that arp limiting client filter with me? We are seeing some new traffic patterns where it looks like user devices are just walking their subnets, and arping for everything -Laramie On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck mailto:cae...@psu.edu>> wrote: We've seen the CPU problem, but I don't think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image002.png@01D79F29.7E132B60] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7Cbdf812520057426b3d3f08d96d614254%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661086771803279%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=CMis8
Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Thanks Brad! We noticed the same. And do the same thing for general connectivity - turn off randomization. The Crestron's, after the app is loaded and you go to screen share, (and all privacy settings are made) the client just times out and says, "unable to connect". Session table indicate no response same with traces. Crestron can reproduce the problem in their lab with their own network. Soo, I'm thinking a bug. But am eagerly awaiting their response. One thing I am struggling with is :Clustering is fun with the google home mini's. EVERY single mini we have on campus has the client is on 1 controller (2 mobility controllers with 1 master) and the airmedia server (other device) is on the opposite controller. We have base level airgroup set up (not where you can invite your friends to your group-I forgot what that is called) That and Crestron's with IOS 14.7 seem to be our issues for this year knock on wood. Ian Cheers Ian J Lyons Network Architect - Rollins College 401.413.1661 Cell 407.628.6396 Desk From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Floyd, Brad Sent: Wednesday, September 1, 2021 11:39 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) * External Email * Ian, iOS 14.0 introduced private MAC addresses. It was broken and devices spoke with both their real MACs and their private MACs. This caused the controllers to blacklist the devices for ARP spoofing. Once the timer expired, the device reconnected again for a while… 14.0 and 14.1 were broken this way. 14.2 fixed it and 14.3 worked fine. I have recently been receiving some tickets for 14.7 and it seems like the bad behavior is back. Our solution to restore connectivity is to have them turn off the private MAC address setting for our SSID(s). Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Ian Lyons Sent: Wednesday, September 1, 2021 10:32 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] Same environment (6.x to 8.x) but no, radius is fine for us. We are on 535's with older 325/303's. (we are AOS 8.6.0.10 which seems -knock on wood -very stable) Our issue is with IOS 14.7 and Crestron Airmedia2's -I suspect the privacy settings are causing us issues. Same Apple hardware ~ 14.3 or earlier is fine. Not to hijack your thread, Ryan. Cheers Ian J Lyons Network Architect - Rollins College 401.413.1661 Cell 407.628.6396 Desk From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Turner, Ryan H mailto:rhtur...@email.unc.edu>> Sent: Wednesday, September 1, 2021 11:27 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) * External Email * This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information c
Re: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
HEy Chuck - would you mind sharing that arp limiting client filter with me? We are seeing some new traffic patterns where it looks like user devices are just walking their subnets, and arping for everything -Laramie On Wed, Sep 1, 2021 at 11:47 AM Enfield, Chuck wrote: > We’ve seen the CPU problem, but I don’t think it resulted in Auth problems > here. It may have and we just missed it because the more severe problems > it caused masked them. > > > > BTW, in our case reducing the amount of ARP calmed the CPU. We applied a > filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless > clients could send and it smoothed out the spikes. > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Cody Ensanian > *Sent:* Wednesday, September 1, 2021 11:41 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall > with large classrooms and delayed connection times (Aruba 8.5.0.13) > > > > I’m hearing issues of high cpu utilization for STM on the controllers > causing issues. Maybe check your controllers and see if you are seeing the > high cpu use for STM. Heard earlier today from our SE that Aruba has > “identified the issue and is working on a fix.” I suggest opening the TAC > case so they can track it better, and help them hone in on a fix better. > We’re seeing the high cpu use on one of our controllers (but this > controller also has higher client load). However, we have not had a flood > of calls to our help desk for wireless issues (not saying they aren’t > happening). Our SE also said if you’re experiencing the issue, disabling > any system or process level debugging as helped, as well as disabling any > SNMP polling. > > > > -Cody > > UCCS > > > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Turner, Ryan H > *Sent:* Wednesday, September 1, 2021 9:27 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* [WIRELESS-LAN] Anyone else seeing any issues in the fall with > large classrooms and delayed connection times (Aruba 8.5.0.13) > > > > This is a stab in the dark. With the University mostly shutdown since the > Spring of 2020 (=not operating in standard mode and most people work from > home), we got campus upgraded from 6.X to 8.X code base. We’ve also > installed many 515 series APs. We are getting a large number of complaints > in large classrooms that connecting to things like eduroam takes a long > time. Looking into the connection, we see many incomplete RADIUS > challenges. The general complaints are ‘we come into the classroom, and > for some folks it can take up to 5 minutes to get connected’. The odd > thing is that our RADIUS infrastructure is very large, polished and load > shared, and we can see no performance issues with any of the RADIUS > servers. We have begun reducing power in the large classrooms to make > association issues better, but so far that hasn’t changed much. We > anticipate opening a ticket with Aruba, soon. We do seem to see the most > complaints in the big classrooms. But I do keep going back to the RADIUS > Challenges incomplete. I know if no reason for those not to complete > unless the connection is broken midway. > > > > Has anyone else seen something like this? > > > > Ryan Turner > > Head of Networking > > Communication Technologies | Information Technology Services > > r...@unc.edu > > +1 919 445 0113 (Office) > > +1 919 274 7926 (Mobile) > > > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7C23c7b4692be5427984b208d96d5eeb66%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661076720957461%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=K%2BCVXnG1IZQhrXjiQ6WZkaNFE7%2FUFb%2F64AyQguElX28%3D=0> > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > <https://
Re: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Same here, enabling arp filtering on the firewall helps greatly. T.J. Norton Wireless Network Architect Network Operations Office: (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since 1971 On Sep 1, 2021, at 11:47 AM, Enfield, Chuck wrote: [ EXTERNAL EMAIL: Do not click any links or open attachments unless you know the sender and trust the content. ] We’ve seen the CPU problem, but I don’t think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I’m hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has “identified the issue and is working on a fix.” I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We’re seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren’t happening). Our SE also said if you’re experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C96e2df1f208e43700aae08d96d5fdb87%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661080757133591%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=zfaQouW6lM30W6dlSHHB8bSX9jxVmqN%2BRdT%2Be6YWMRQ%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C96e2df1f208e43700aae08d96d5fdb87%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661080757143589%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We've seen the CPU problem, but I don't think it resulted in Auth problems here. It may have and we just missed it because the more severe problems it caused masked them. BTW, in our case reducing the amount of ARP calmed the CPU. We applied a filter (Thank you Colin Joseph.) to limit the amount of ARP our wireless clients could send and it smoothed out the spikes. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Cody Ensanian Sent: Wednesday, September 1, 2021 11:41 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F27.2E913C50] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7C23c7b4692be5427984b208d96d5eeb66%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661076720957461%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=K%2BCVXnG1IZQhrXjiQ6WZkaNFE7%2FUFb%2F64AyQguElX28%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7C23c7b4692be5427984b208d96d5eeb66%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661076720967461%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=tPFZ2xToscFSeEwuMMFC6j%2BUh4yzENSUnwYXwtWFvyE%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We have seen issues with the Lenovo vantage software causing to many ARPs, on top of the IOS 14 issues, that caused the STM process to crash as well. ARP inspection helped mitigate this quite a bit, but would randomly crash one of our controller due a select few clients hashing there. Ryan - Does CPU load reflect high on any of the controllers or seeing anything in the logs system wise? We recently worked with securew2 to help automate the removal of the Lenovo software in the via the client as part of on boarding. T.J. Norton Wireless Network Architect Network Operations Office: (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since 1971 On Sep 1, 2021, at 11:40 AM, Turner, Ryan H wrote: [ EXTERNAL EMAIL: Do not click any links or open attachments unless you know the sender and trust the content. ] Its been a while since I look at that. Would be a good path to check Thank you. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Michael Davis Sent: Wednesday, September 1, 2021 11:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Is your backend (controllers - Radius) all jumbo frame clean? We've seen issues with large EAP-TLS packets getting fragmented. We also had a specific OS8 release bug affecting AP-515s specifically, but it seems like we're in perpetual bug-chasing mode so I can't recall what version that was. (Probably 8.5 something) (edit: I just saw the 8.5.0.13 in the subject. You may have to move away from that..) On 9/1/21 11:27 AM, Turner, Ryan H wrote: This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C8afdf913cc1c457c9aa408d96d5ec1ad%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661076026513640%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=bY9b5S2XPz1w92U0%2FDXg1Z4ymYpHBDqVExd9J%2Fw0ivo%3D=0> -- Mike Davis IT - University of Delaware - 302.831.8756 Newark, DE 19716 Email da...@udel.edu<mailto:da...@udel.edu> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7C8afdf913cc1c457c9aa408d96d5ec1ad%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661076026513640%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=bY9b5S2XPz1w92U0%2FDXg1Z4ymYpHBDqVExd9J%2Fw0ivo%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply.
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We're not having any unusual problems now, but we have in the past. Two suggestions I can offer are: * Search your controller syslog for "Authentication server request timeout". This will tell you if the controllers are sending auth requests and not getting replies back. We've had this happen when RDAIUS servers report being fat and happy. Best explanation I can offer is that VMs sometimes lie. * Check the controller 802.1X counters to make sure they're not throttling authentications. https://community.arubanetworks.com/blogs/ssasi1/2020/10/28/how-does-auth-throttling-feature-work-and-what-are-the-associated-cli-commands. If this does occur, it tends to happen at times of high user mobility. Good luck. Chuck From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 11:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ccae104%40PSU.EDU%7Cbf80edde3d5a412daf6108d96d5cfb72%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661068387237277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=HBe5V3IJtEi%2FhbE4qSWswW3PQepfmQCI1Hn1Q%2Fi%2BCm8%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
I'm hearing issues of high cpu utilization for STM on the controllers causing issues. Maybe check your controllers and see if you are seeing the high cpu use for STM. Heard earlier today from our SE that Aruba has "identified the issue and is working on a fix." I suggest opening the TAC case so they can track it better, and help them hone in on a fix better. We're seeing the high cpu use on one of our controllers (but this controller also has higher client load). However, we have not had a flood of calls to our help desk for wireless issues (not saying they aren't happening). Our SE also said if you're experiencing the issue, disabling any system or process level debugging as helped, as well as disabling any SNMP polling. [cid:image001.png@01D79F14.D04A94F0] -Cody UCCS From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Turner, Ryan H Sent: Wednesday, September 1, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Its been a while since I look at that. Would be a good path to check Thank you. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Michael Davis Sent: Wednesday, September 1, 2021 11:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Is your backend (controllers - Radius) all jumbo frame clean? We've seen issues with large EAP-TLS packets getting fragmented. We also had a specific OS8 release bug affecting AP-515s specifically, but it seems like we're in perpetual bug-chasing mode so I can't recall what version that was. (Probably 8.5 something) (edit: I just saw the 8.5.0.13 in the subject. You may have to move away from that..) On 9/1/21 11:27 AM, Turner, Ryan H wrote: This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community -- Mike Davis IT - University of Delaware - 302.831.8756 Newark, DE 19716 Email da...@udel.edu<mailto:da...@udel.edu> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] [External] [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
So I should say that while I dropped the 515, most of these classrooms have 300 series. So that part isn’t generally related (or consistent to one model type). Ryan From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Norton, Thomas (Network Operations) Sent: Wednesday, September 1, 2021 11:36 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We’re on 8.6.0.11 and not seeing any issues currently, but also running 225/325s in the majority of our class rooms. We just purchased our first round 5xx access points and two of our LPVs are rung 535, 577, and 534s without issue on 8.7.0.4 Do you guys have the HE bit disabled? I know the 515s also have quite few bugs still in play right now. T.J. Norton Wireless Network Architect Network Operations Office: (434) 592-6552 [cid:~WRD0001.jpg] Liberty University | Training Champions for Christ since 1971 On Sep 1, 2021, at 11:27 AM, Turner, Ryan H wrote: [ EXTERNAL EMAIL: Do not click any links or open attachments unless you know the sender and trust the content. ] This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7Caee2f6f05756443b246b08d96d5cfb63%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661068386827653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=P2Cf3gTkKGGk0cUQO6Mcc%2FbVtHDYRnuaNXrodogXQD8%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Ian, iOS 14.0 introduced private MAC addresses. It was broken and devices spoke with both their real MACs and their private MACs. This caused the controllers to blacklist the devices for ARP spoofing. Once the timer expired, the device reconnected again for a while... 14.0 and 14.1 were broken this way. 14.2 fixed it and 14.3 worked fine. I have recently been receiving some tickets for 14.7 and it seems like the bad behavior is back. Our solution to restore connectivity is to have them turn off the private MAC address setting for our SSID(s). Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Ian Lyons Sent: Wednesday, September 1, 2021 10:32 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] Same environment (6.x to 8.x) but no, radius is fine for us. We are on 535's with older 325/303's. (we are AOS 8.6.0.10 which seems -knock on wood -very stable) Our issue is with IOS 14.7 and Crestron Airmedia2's -I suspect the privacy settings are causing us issues. Same Apple hardware ~ 14.3 or earlier is fine. Not to hijack your thread, Ryan. Cheers Ian J Lyons Network Architect - Rollins College 401.413.1661 Cell 407.628.6396 Desk From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Turner, Ryan H mailto:rhtur...@email.unc.edu>> Sent: Wednesday, September 1, 2021 11:27 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) * External Email * This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Cilyons%40ROLLINS.EDU%7Ce2e13ef2ff654bab861708d96d5cfb60%7Cb8e8d71a947d41dd81dd8401dcc51007%7C0%7C0%7C637661068384958917%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=oIZc8yvMlX8TyJuTUk0ZTRxKNSK%2BGUgco9fdKVA1vJA%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Is your backend (controllers - Radius) all jumbo frame clean? We've seen issues with large EAP-TLS packets getting fragmented. We also had a specific OS8 release bug affecting AP-515s specifically, but it seems like we're in perpetual bug-chasing mode so I can't recall what version that was. (Probably 8.5 something) (edit: I just saw the 8.5.0.13 in the subject. You may have to move away from that..) On 9/1/21 11:27 AM, Turner, Ryan H wrote: This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu <mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community -- Mike Davis IT - University of Delaware - 302.831.8756 Newark, DE 19716 Email da...@udel.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [External] [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
We’re on 8.6.0.11 and not seeing any issues currently, but also running 225/325s in the majority of our class rooms. We just purchased our first round 5xx access points and two of our LPVs are rung 535, 577, and 534s without issue on 8.7.0.4 Do you guys have the HE bit disabled? I know the 515s also have quite few bugs still in play right now. T.J. Norton Wireless Network Architect Network Operations Office: (434) 592-6552 [http://www.liberty.edu/media/1616/40themail/wordmark-for-email.jpg] Liberty University | Training Champions for Christ since 1971 On Sep 1, 2021, at 11:27 AM, Turner, Ryan H wrote: [ EXTERNAL EMAIL: Do not click any links or open attachments unless you know the sender and trust the content. ] This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctnorton7%40LIBERTY.EDU%7Caee2f6f05756443b246b08d96d5cfb63%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C637661068386827653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=P2Cf3gTkKGGk0cUQO6Mcc%2FbVtHDYRnuaNXrodogXQD8%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
Same environment (6.x to 8.x) but no, radius is fine for us. We are on 535's with older 325/303's. (we are AOS 8.6.0.10 which seems -knock on wood -very stable) Our issue is with IOS 14.7 and Crestron Airmedia2's -I suspect the privacy settings are causing us issues. Same Apple hardware ~ 14.3 or earlier is fine. Not to hijack your thread, Ryan. Cheers Ian J Lyons Network Architect - Rollins College 401.413.1661 Cell 407.628.6396 Desk From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Turner, Ryan H Sent: Wednesday, September 1, 2021 11:27 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) * External Email * This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We’ve also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are ‘we come into the classroom, and for some folks it can take up to 5 minutes to get connected’. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn’t changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Cilyons%40ROLLINS.EDU%7Ce2e13ef2ff654bab861708d96d5cfb60%7Cb8e8d71a947d41dd81dd8401dcc51007%7C0%7C0%7C637661068384958917%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=oIZc8yvMlX8TyJuTUk0ZTRxKNSK%2BGUgco9fdKVA1vJA%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13)
This is a stab in the dark. With the University mostly shutdown since the Spring of 2020 (=not operating in standard mode and most people work from home), we got campus upgraded from 6.X to 8.X code base. We've also installed many 515 series APs. We are getting a large number of complaints in large classrooms that connecting to things like eduroam takes a long time. Looking into the connection, we see many incomplete RADIUS challenges. The general complaints are 'we come into the classroom, and for some folks it can take up to 5 minutes to get connected'. The odd thing is that our RADIUS infrastructure is very large, polished and load shared, and we can see no performance issues with any of the RADIUS servers. We have begun reducing power in the large classrooms to make association issues better, but so far that hasn't changed much. We anticipate opening a ticket with Aruba, soon. We do seem to see the most complaints in the big classrooms. But I do keep going back to the RADIUS Challenges incomplete. I know if no reason for those not to complete unless the connection is broken midway. Has anyone else seen something like this? Ryan Turner Head of Networking Communication Technologies | Information Technology Services r...@unc.edu<mailto:r...@unc.edu> +1 919 445 0113 (Office) +1 919 274 7926 (Mobile) ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba AirSlice?
Hello All, Has anyone implemented Aruba AirSlice? What has been your experience with that feature after it was activated? Thanks in advance. -- [image: Santa Clara University] <https://www.scu.edu/> *Antonio Garcia* Network Engineer Cyberinfrastructure Technologies 500 El Camino Real, Santa Clara, CA 95053 phone | 408-554-5531 email | aagar...@scu.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: Aruba Central for Managing IAP Feedback
Chad, We started using Aruba Central in production about a year ago, and it's gone well. What I like: * 100% cloud-based * Pretty Good UI * Very smooth onboarding/setup (got eduroam running in <1h) * Virtual Controller software (runs on an elected AP) seems to be rock-solid * Does a good job of supporting and handling multicast discovery protocols * Automatic channel/broadcast strength selection works well What I wish was better: * No programmatic access to configuration settings (would love to change PSKs by API), unless you want to do it ALL via API. * the Aruba Virtual Controllers (running on the AP) don't currently support giant flat L2 networks -- we do L3 zones per-building, which means L3 roams when clients move between buildings * When troubleshooting in the UI, data doesn't update instantaneously. The live view is pretty quick, but not like having an on-prem controller * It's not cheap #end [https://www.cedarville.edu/images/default-source/email/2column-cu.png?ver=20210720] Nat Biggs Network Analyst Information Technology Adjunct Professor School of Business Administration Cedarville University o: 937-766-7905 cedarville.edu<https://cedarville.edu/> [https://www.cedarville.edu/images/default-source/email/2column-tagline.png?ver=20210720] [https://www.cedarville.edu/images/default-source/email/email_twitter-22px.png?ver=20210720]<https://twitter.com/cedarville> [https://www.cedarville.edu/images/default-source/email/email_youtube-22px.png?ver=20210720] <https://www.youtube.com/user/cedarvilleu> [https://www.cedarville.edu/images/default-source/email/email_facebook-22px.png?ver=20210720] <https://www.facebook.com/cedarville> [https://www.cedarville.edu/images/default-source/email/email_instagram-22px.png?ver=20210720] <https://www.instagram.com/cedarville/> From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Street, Chad A Sent: Wednesday, August 4, 2021 2:10 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Aruba Central for Managing IAP Feedback Hello fellow wireless minions, for anyone currently using Aruba Central to manage IAPs, could you kindly provide feedback on your experience. Pros, Cons, satisfaction with the user interface, oddities and any other interesting experiences you have to share would be most welcome. Feel free to email me privately below. Best regards, Chad Street Emory University and Healthcare chad.str...@emory.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Cnbiggs112%40cedarville.edu%7Cd2513c605f2745248d6408d957733a08%7C81c32413015d4ba8a93be1c28e355738%7C1%7C0%7C637636974691223515%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=iQ2uwP7j%2FaMGuF4XYrX0dXMWz8Nbuw0ubtelHWDujD4%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba Central for Managing IAP Feedback
Hello fellow wireless minions, for anyone currently using Aruba Central to manage IAPs, could you kindly provide feedback on your experience. Pros, Cons, satisfaction with the user interface, oddities and any other interesting experiences you have to share would be most welcome. Feel free to email me privately below. Best regards, Chad Street Emory University and Healthcare chad.str...@emory.edu ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: Aruba and SAML SSO
That is interesting Tim, let me investigate this further as this is new news to me... Thx From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Tim Cappalli Sent: 26 July 2021 15:19 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba and SAML SSO CPPM will parse out the SAML assertion attributes as long as you add them to the SSO dictionary in CPPM. You can then use them in role mapping or enforcement in an application authorization service. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Martin MacLeod-Brown mailto:mmacl...@london.edu>> Sent: Monday, July 26, 2021 10:13:15 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [WIRELESS-LAN] Aruba and SAML SSO Hi Everyone Just reaching out here to see if anyone has managed this using Aruba technologies? We have a B2B client who enrols onto one of our Open Courses, using an email address of their choice. We capture that email address in AAD and they will be sent an invite to join the relevant Teams/O365 resources that apply to them and to reset their initial password. When these clients arrive at campus they connect to our guest Wi-Fi where they self register via our Captive Portal Is there a way that they can use their B2B details that they signed up with originally to log into the guest Wi-Fi? I know last time I looked at this, I could get Clearpass and AAD talking however the authentication token that AAD was sending back after a successful login was just some simple hashed text and I couldn't work out how to intercept that or craft a service/role around it. Has anyone done something like this? Martin ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctim.cappalli%40MICROSOFT.COM%7C020c7434966448c9dadf08d9503f969b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637629056568170328%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=r5U%2FzUFlS3udFdIU5eXCoCD3GBoYrg%2FoyzttBfz%2BRuQ%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://www.educause.edu/community> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: Aruba and SAML SSO
CPPM will parse out the SAML assertion attributes as long as you add them to the SSO dictionary in CPPM. You can then use them in role mapping or enforcement in an application authorization service. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Martin MacLeod-Brown Sent: Monday, July 26, 2021 10:13:15 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Aruba and SAML SSO Hi Everyone Just reaching out here to see if anyone has managed this using Aruba technologies? We have a B2B client who enrols onto one of our Open Courses, using an email address of their choice. We capture that email address in AAD and they will be sent an invite to join the relevant Teams/O365 resources that apply to them and to reset their initial password. When these clients arrive at campus they connect to our guest Wi-Fi where they self register via our Captive Portal Is there a way that they can use their B2B details that they signed up with originally to log into the guest Wi-Fi? I know last time I looked at this, I could get Clearpass and AAD talking however the authentication token that AAD was sending back after a successful login was just some simple hashed text and I couldn’t work out how to intercept that or craft a service/role around it. Has anyone done something like this? Martin ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=04%7C01%7Ctim.cappalli%40MICROSOFT.COM%7C020c7434966448c9dadf08d9503f969b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637629056568170328%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000=r5U%2FzUFlS3udFdIU5eXCoCD3GBoYrg%2FoyzttBfz%2BRuQ%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba and SAML SSO
Hi Everyone Just reaching out here to see if anyone has managed this using Aruba technologies? We have a B2B client who enrols onto one of our Open Courses, using an email address of their choice. We capture that email address in AAD and they will be sent an invite to join the relevant Teams/O365 resources that apply to them and to reset their initial password. When these clients arrive at campus they connect to our guest Wi-Fi where they self register via our Captive Portal Is there a way that they can use their B2B details that they signed up with originally to log into the guest Wi-Fi? I know last time I looked at this, I could get Clearpass and AAD talking however the authentication token that AAD was sending back after a successful login was just some simple hashed text and I couldn't work out how to intercept that or craft a service/role around it. Has anyone done something like this? Martin ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba AP 510 Series/Windows Devices
Yes, this was AP manufacturer independent as the root cause of the issue was in the way the device driver handles 802.11ax (WiFi 6). If the user has updated their driver the issue is resolved, but you’d have to rely on the user to properly upgrade their devices. If all they connect to is wireless, they can’t update the device driver. To disable the High Efficiency Radio, Go to Configure -> System -> Profiles -> Wireless LAN Select the High-Efficiency Radio and disable Aruba WebHelp Doc https://www.arubanetworks.com/techdocs/ArubaOS_8.8.0_Web_Help/Content/arubaos-solutions/virtual-ap/high-effi-aps.htm Or go to the AP Group, then at the top right drop down your username, select Preferences, enable Show advanced profiles. Select You AP Group, and click profiles expand RF Management Expand 5GHz radio, and update the radio profile you’re using for the AP Group. From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Ronald Loneker Sent: Tuesday, June 8, 2021 12:39 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Aruba AP 510 Series/Windows Devices Hi Everyone - With the limited number of Aruba IAP 515 devices we have remaining, we've been upgrading a number of buildings to this device model from our old legacy Arubas. Back in January, I came across a thread about Aruba AP 510 series and conflicts with a version of an Intel driver I thought on this list for its high efficiency profile setting that we needed to disable to avoid wireless issues. I can't seem to find the thread, however. Has anyone noticed this issue anymore with this series of Arubas? I'm not sure whether we still should keep this profile setting off or we should enable it to take more advantage of its functionality. Most of our students are now off campus so testing it right now isn't really an option. Any thoughts from your experiences? Ron Loneker, Jr. Director, IT Special Projects Saint Elizabeth University Mahoney Library 2 Convent Road Morristown, NJ 07960 Phone: 973-290-4229 e-mail: rlone...@steu.edu<mailto:rlone...@steu.edu> Saint Elizabeth University's IT department will never ask for your password, social security number or other personal information in an e-mail message. Please do not share any information with others! ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.educause.edu%2fcommunity=E,1,9NMIZr-mqFF7nq3QkFKhBVFi_vveL26rbK2PN1mSbOJUohNO6O6WYJHHiW43D_IOvDymoSllJ_SWFvRB4X90_i5CyIk9XJUGz1h09Zw8VhnNOk34ULa_YJCKrQ,,=1> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba AP 510 Series/Windows Devices
Hi Everyone - With the limited number of Aruba IAP 515 devices we have remaining, we've been upgrading a number of buildings to this device model from our old legacy Arubas. Back in January, I came across a thread about Aruba AP 510 series and conflicts with a version of an Intel driver I thought on this list for its high efficiency profile setting that we needed to disable to avoid wireless issues. I can't seem to find the thread, however. Has anyone noticed this issue anymore with this series of Arubas? I'm not sure whether we still should keep this profile setting off or we should enable it to take more advantage of its functionality. Most of our students are now off campus so testing it right now isn't really an option. Any thoughts from your experiences? Ron Loneker, Jr. Director, IT Special Projects Saint Elizabeth University Mahoney Library 2 Convent Road Morristown, NJ 07960 Phone: 973-290-4229 e-mail: rlone...@steu.edu *Saint Elizabeth University's IT department will never ask for your password, social security number or other personal information in an e-mail message.* *Please do not share any information with others!* ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] Aruba Founders Retiring (CEO, CTO, and Architect)
On Jun 2, 2021, at 08:09, Johnson, Neil M mailto:neil-john...@uiowa.edu>> wrote: CEO Blog Post - https://blogs.arubanetworks.com/uncategorized/lets-name-it-aruba/<https://urldefense.com/v3/__https://blogs.arubanetworks.com/uncategorized/lets-name-it-aruba/__;!!Dq0X2DkFhyF93HkjWTBQKhk!Cnb1xiQgrLAThmoMVkpkQjNK1vo-w0syYItoUkmHoX9vnNxMt3pVcAfIkRIxy_UvBQXOEw$> CTO Blog Post - https://blogs.arubanetworks.com/uncategorized/my-aruba-journey/<https://urldefense.com/v3/__https://blogs.arubanetworks.com/uncategorized/my-aruba-journey/__;!!Dq0X2DkFhyF93HkjWTBQKhk!Cnb1xiQgrLAThmoMVkpkQjNK1vo-w0syYItoUkmHoX9vnNxMt3pVcAfIkRIxy_UwO0oztA$> Chief Architect - https://blogs.arubanetworks.com/uncategorized/reflecting-on-my-aruba-family/<https://urldefense.com/v3/__https://blogs.arubanetworks.com/uncategorized/reflecting-on-my-aruba-family/__;!!Dq0X2DkFhyF93HkjWTBQKhk!Cnb1xiQgrLAThmoMVkpkQjNK1vo-w0syYItoUkmHoX9vnNxMt3pVcAfIkRIxy_UkeABHnQ$> Thanks for sharing, Neil. Definitely end of an era. Fingers crossed that this transition is as successful as their last few major transitions over the past few years! -- Julian Y. Koh Associate Director, Telecommunications and Network Services Northwestern Information Technology 2020 Ridge Avenue #331 Evanston, IL 60208 +1-847-467-5780 Northwestern IT Web Site: <https://www.it.northwestern.edu/> PGP Public Key: <https://bt.ittns.northwestern.edu/julian/pgppubkey.html> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Aruba Founders Retiring (CEO, CTO, and Architect)
CEO Blog Post - https://blogs.arubanetworks.com/uncategorized/lets-name-it-aruba/ CTO Blog Post - https://blogs.arubanetworks.com/uncategorized/my-aruba-journey/ Chief Architect - https://blogs.arubanetworks.com/uncategorized/reflecting-on-my-aruba-family/ -Neil -- Neil Johnson (he/him/his) Network Architect, ITS Network Engineering Services 5 Lindquist Center South, Iowa City, Iowa 52242 Office: 319-384-0938 uiowa.edu<http://www.uiowa.edu/> [The University of Iowa Logo] ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
Chris, Well, I have a mix of problems. Since I moved to a MM dual controller scenario, roaming for students stopped working properly at 8.7.1.0. So I upgraded to 8.7.1.3 and that seemed to mostly fix it though there are some delays when going to a new ip address segment. Those delays didn’t exist for me in 6.8.x.x but the delays are only 30 to 60 seconds so livable for the moment. I didn’t have these delays when I was on a single controller with the same configuration. I will fix that this summer when I make the IP’s follow everyone from zone to zone which should get rid of this roaming issue. But lately I have had a few complaints where they say wireless went away on them. I don’t see evidence in Airwave. I have only been on 8.7.1.3 for about 2 weeks now so these complaints make me paranoid. Some of these complaints are from our own tech team. When I go in to test, I don’t observe it. I am hopefully going to meet with one of them today to see why they keep having issues with what they think is poor signal quality when it is perfect for me in the same location. Before I moved to Mobility Manager, I had maybe 2 wireless complaints a semester. After moving from 6.x.x.x to 8.7.x.x in MM complaints shot through the roof for me. Aruba helped fine tune some of this for me and upgrading to 8.7.1.3 seems a lot more stable, but I am only working on 2 weeks of it so far. If I had to do over, I would have gone to the latest 8.6 version instead. If I learn anything more today, I will post again. Tim *From:* The EDUCAUSE Wireless Issues Community Group Listserv [mailto: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] *On Behalf Of *Johnson, Christopher *Sent:* Friday, May 21, 2021 10:12 AM *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues Tim, I'm curious what kind of issues you're seeing? We predominately have 225s, 205Hs and 205s - so I'm interested a bit more in what you're seeing. I second the thanks again for all the great feedback/experiences on ArubaOS Versions everyone is sharing. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook and Twitter -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv < WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Rob Harris Sent: Thursday, May 20, 2021 8:49 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv < WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [ mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU ] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 5
RE: [WIRELESS-LAN] Aruba 8.7 issues
Tim, I'm curious what kind of issues you're seeing? We predominately have 225s, 205Hs and 205s - so I'm interested a bit more in what you're seeing. I second the thanks again for all the great feedback/experiences on ArubaOS Versions everyone is sharing. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook and Twitter -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Rob Harris Sent: Thursday, May 20, 2021 8:49 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7C%7C2d907adbc2584c28faf508d91b960272%7C085f983a0b694270b71d10695076bafe%7C1%7C0%7C637571153364879263%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=W8HPykdxHtmy0S2r89G7DwCda2E7AIaVCMym66VzJTA%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7C%7C2d907adbc2584c28faf508d91b960272%7C085f983a0b694270b71d10695076bafe%7C1%7C0%7C637571153364889258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=Ir9W44SdIGc1%2FvP2wcOTvX0NdK2W6KlUYw%2B%2BFz%2BrzT4%3Dreserved=0 **
RE: [WIRELESS-LAN] Aruba 8.7 issues
The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7Ce5b7bf5d1d62433f38fd08d91b94654f%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637571146437877600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=8KAYrcvzPrK3vYTrs23SBjCuXxTXDUYRrjSmiT01mBQ%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7Ce5b7bf5d1d62433f38fd08d91b94654f%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637571146437882581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=FAcuaBHmraK%2F3N2fPaOuwJFgG6kfZpPIyoZMLZ1sLPg%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
8.6.0.9 is the latest "conservative" release (what should be their most stable go-to release). Just be sure to check what your environment's min requirements are. For example... for AP-505H, the min software release needed is 8.7.0.0. It's these min req's that usually make you have to go beyond what the latest conservative release is (our case, since we are using 505Hs) Cody -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 7:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? > We’re looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) – from > this thread I’m getting a strong “Do Not Engage” vibe. But interested > in everyone’s thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? > We’re looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) – from > this thread I’m getting a strong “Do Not Engage” vibe. But interested > in everyone’s thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [EXT] Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We upgraded to 8.7 when we deployed some 575’s. Had a bunch of client issues and a weird issues with 535’s classifying neighboring ap’s as interfering rouges with 8.7.1.1 and 8.7.1.2. I would recommend skipping 8.7.1.1 and 8.7.1.2.8.7.1.3 has been better. Kevin Grover Utah State University From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Iverson, Jeremy Date: Wednesday, May 19, 2021 at 8:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [EXT] Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues We too are looking to deploy v8.7.1.3 to support some new Ap503H’s. I wish it was a conservative release, but glad some are not having any issues. Thanks to those who have shared your experiences! Thanks, Jeremy Iverson Northern State University From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Martin Reynolds Sent: Wednesday, May 19, 2021 8:39 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues CAUTION: This email originated from outside of NSU. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username and password. We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion mailto:pertalio...@appstate.edu>> wrote: We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: So how’s the ArubaOS 8.7 code train treating everyone these days? We’re looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking at a couple AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m getting a strong “Do Not Engage” vibe. But interested in everyone’s thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://www.facebook.com/ISUITHelp/> and Twitter<https://twitter.com/ISUITHelp> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all sta
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We too are looking to deploy v8.7.1.3 to support some new Ap503H’s. I wish it was a conservative release, but glad some are not having any issues. Thanks to those who have shared your experiences! Thanks, Jeremy Iverson Northern State University From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Martin Reynolds Sent: Wednesday, May 19, 2021 8:39 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues CAUTION: This email originated from outside of NSU. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username and password. We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion mailto:pertalio...@appstate.edu>> wrote: We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: So how’s the ArubaOS 8.7 code train treating everyone these days? We’re looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking at a couple AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m getting a strong “Do Not Engage” vibe. But interested in everyone’s thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://www.facebook.com/ISUITHelp/> and Twitter<https://twitter.com/ISUITHelp> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all stay up. I breathe a bit easier. It appears that in the process of trying to figure out the issues we were seeing, we kept turning up the logging level, which increased the amount of cpu the controller had to spend on logging, to the detriment of processing packets for communication to access points. We still see a few issues with communication between the ap's and the controllers, but now, at least the ap's remain up on the redundant tunnels. Aruba is still working on resolving a load issue on the controller where it's dropping packets. Robert Spellm
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion wrote: > > We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments > planned this summer. > > As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to > 8.7.1.3. We had no issues during this process. > > At this point, we have about 75 AP-503H units deployed and haven't had any > issues with the devices booting up and becoming active on the network. The > rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. > No crashes or lost units since the upgrade. > > > > On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher > wrote: > >> So how’s the ArubaOS 8.7 code train treating everyone these days? We’re >> looking at doing some maintenance here shortly and moving from 8.5.0.11 to >> 8.6 code train for some mini OS enhancements – and looking at a couple >> AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m >> getting a strong “Do Not Engage” vibe. But interested in everyone’s >> thoughts given the additional few months that have passed since then? >> >> *Christopher Johnson* >> Wireless Network Engineer >> Office of Technology Solutions | Illinois State University >> (309) 438-8444 >> >> Stay connected with ISU IT news and tips with @ISU IT Help on Facebook >> <https://www.facebook.com/ISUITHelp/> and Twitter >> <https://twitter.com/ISUITHelp> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Robert Spellman >> *Sent:* Wednesday, December 30, 2020 8:41 PM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues >> >> >> >> *[This message came from an external source. If suspicious, report to >> ab...@ilstu.edu ] * >> >> Good news, all of our access points are up tonight. Bad news, it was a >> rough couple of days before we got to this point. A lot of zoom calls with >> Aruba engineers. >> >> >> >> Early on in the debug process, I noticed logs from the access points >> indicating lost packets, duplicate packets, and packets out of sequence. >> Tunnels between the ap's and the controllers weren't stable. Ap's were >> taking hours to boot, or never coming up at all. Ap's that were up and >> passing traffic would drop from the network. >> >> >> >> The first engineer at Aruba said it looks like a network issue, and we >> should look into the network switches between the ap's and the >> controllers. The engineer turned up logging, rebooted controllers and >> access points, and nothing seemed to keep the ap's online for long. I had >> performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the >> controllers back to 8.5.0.3, and this didn't resolve the issue. >> >> >> >> Three more days of tech support. By now, we have upgraded to 8.7.0.0, >> changed logging some more, and collected lots of log files. I install >> devices on both switches which are capturing packets, and we can clearly >> see that all packets from the ap's are arriving properly at the controller, >> which is discarding them. I think we can finally stop blaming the >> network. At this point, we are beyond level one tech support, and >> yesterday, even had developers on the zoom call with us. Then one engineer >> says who turned all this logging on, and turns it all off. Within five >> minutes, all access points are back online. >> >> >> >> We reboot all of the ap's, and within five minutes, they are all back. >> We watch for a few hours, and they all stay up. I breathe a bit easier. >> >> >> >> It appears that in the process of trying to figure out the issues we were >> seeing, we kept turning up the logging level, which increased the amount of >> cpu the controller had to spend on logging, to the detriment of processing >> packets for communication to access points. >> >> >> >> We still see a few issues with communication between the ap's and the >> controllers, but now, at least the ap's remain up on the redundant >> tunnels. Aruba is still working on resolving a load issue on the >> controller where it's dropping packets. >> >> >> >> Robert Spellman >> >> *Associate Director for Network Services* >> >
Re: [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? We’re > looking at doing some maintenance here shortly and moving from 8.5.0.11 to > 8.6 code train for some mini OS enhancements – and looking at a couple > AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m > getting a strong “Do Not Engage” vibe. But interested in everyone’s > thoughts given the additional few months that have passed since then? > > *Christopher Johnson* > Wireless Network Engineer > Office of Technology Solutions | Illinois State University > (309) 438-8444 > > Stay connected with ISU IT news and tips with @ISU IT Help on Facebook > <https://www.facebook.com/ISUITHelp/> and Twitter > <https://twitter.com/ISUITHelp> > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Robert Spellman > *Sent:* Wednesday, December 30, 2020 8:41 PM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues > > > > *[This message came from an external source. If suspicious, report to > ab...@ilstu.edu ] * > > Good news, all of our access points are up tonight. Bad news, it was a > rough couple of days before we got to this point. A lot of zoom calls with > Aruba engineers. > > > > Early on in the debug process, I noticed logs from the access points > indicating lost packets, duplicate packets, and packets out of sequence. > Tunnels between the ap's and the controllers weren't stable. Ap's were > taking hours to boot, or never coming up at all. Ap's that were up and > passing traffic would drop from the network. > > > > The first engineer at Aruba said it looks like a network issue, and we > should look into the network switches between the ap's and the > controllers. The engineer turned up logging, rebooted controllers and > access points, and nothing seemed to keep the ap's online for long. I had > performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the > controllers back to 8.5.0.3, and this didn't resolve the issue. > > > > Three more days of tech support. By now, we have upgraded to 8.7.0.0, > changed logging some more, and collected lots of log files. I install > devices on both switches which are capturing packets, and we can clearly > see that all packets from the ap's are arriving properly at the controller, > which is discarding them. I think we can finally stop blaming the > network. At this point, we are beyond level one tech support, and > yesterday, even had developers on the zoom call with us. Then one engineer > says who turned all this logging on, and turns it all off. Within five > minutes, all access points are back online. > > > > We reboot all of the ap's, and within five minutes, they are all back. We > watch for a few hours, and they all stay up. I breathe a bit easier. > > > > It appears that in the process of trying to figure out the issues we were > seeing, we kept turning up the logging level, which increased the amount of > cpu the controller had to spend on logging, to the detriment of processing > packets for communication to access points. > > > > We still see a few issues with communication between the ap's and the > controllers, but now, at least the ap's remain up on the redundant > tunnels. Aruba is still working on resolving a load issue on the > controller where it's dropping packets. > > > > Robert Spellman > > *Associate Director for Network Services* > > Information and Library Services > > *Bates College* > > p: > > 207-786-6422 > > a: > > 110 Russell Street, Lewiston, ME 04240 > > w: > > www.bates.edu e: rspell...@bates.edu > > > > > > On Tue, Dec 29, 2020 at 8:37 AM Robert Spellman wrote: > > Our latest purchase of Aruba access points included some that required > 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas > break. We have three 7220 controllers and a virtual mobility master > running, with around 1200 access points. > > > > Thursday morning, we did the upgrade on the master and the three > controllers. After a re
Re: [WIRELESS-LAN] Aruba 8.7 issues
On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? We’re > looking at doing some maintenance here shortly and moving from 8.5.0.11 > to 8.6 code train for some mini OS enhancements – and looking at a > couple AP-575 APs (which of course requires 8.7 minimum) – from this > thread I’m getting a strong “Do Not Engage” vibe. But interested in > everyone’s thoughts given the additional few months that have passed > since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
