Sorry I’m late to the party. I got ahold of a great tech at Aruba last night. We finished up with his suggested work-arounds 2-3 hours before the advisory came out. Implemented the client rebalancing threshold change, also at the seemingly popular 15% Turned on ALL the broadcast/multicast optimization switches, at both the VLAN and virtual-ap level. Tweaked the mentioned ACLs (but I doubt that did much. An ACL would just prevent sending the traffic, not wasting cycles generating the traffic.) Tech mentioned SNMP polling could be an issue, but since we’re preferring AMON we left that one alone. Lastly, restarted STM process on each controller. Definitely service impacting but staggering the restart across multiple controllers seemed to help. I hadn’t seen a few of those in the thread.
Regards, Colin Colin Randall Manager of Data Networking Information and Technology Solutions (ITS) 1600 Jackson Street, Suite 360 303-384-2208 | crand...@mines.edu From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of "Enfield, Chuck" <cae...@psu.edu> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Date: Thursday, September 2, 2021 at 11:06 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) CAUTION: This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. I’d like to suggest sending them home, but if we learned anything last year it’s that home wi-fi isn’t so great either. How many times have you heard, “It works when I’m at home?” Well now we know, not always. From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Floyd, Brad Sent: Thursday, September 2, 2021 1:00 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) JD, If we wrap each of them along with their devices in an aluminum foil bubble, each user would have their own collision domain. The MIMO reflections would be awesome, we wouldn’t need more than a single channel architecture, and any channel contention would be self-imposed. Here’s hoping we get to catch up again at the next post-COVID WLPC. Thanks, Brad From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Davis, Jonathan Alan Sent: Thursday, September 2, 2021 11:42 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) [EXTERNAL SENDER] “That's been my experience for years. The network works great when there are no students around. My working theory is that students emit RF interference, but research ethics won’t let me run the tests, so we'll never know for sure.” It’s worse than that! They are walking bags of water which absorb the good RF, and their devices transmit the bad RF! It’s a conspiracy I tell ya! We’re going to work with TAC on capturing traffic during a class that is known to have issues. After that, we plan to change the rebalancing threshold as well. Thanks everyone for the feedback! JD From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Enfield, Chuck <cae...@psu.edu> Date: Thursday, September 2, 2021 at 12:15 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) I will also add that our problems did not increase linearly with client count on a controller. Below 5K there was no user impact. Around 5K problems started and the severity increased quickly. I doubt there’s anything magic about 5K, and the threshold will be different on every network based on a variety of implementation details, but I’d expect that pattern to be common. From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Enfield, Chuck Sent: Thursday, September 2, 2021 11:21 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Between 5k and 6k clients on a 7240xm is where we started seeing problems. Lighter loaded controllers were OK. From: "Street, Chad A" <cstr...@emory.edu> Sent: Thursday, September 2, 2021 11:03 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) We are a balanced cluster, notes about load below: "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Rob Harris <robert.har...@culinary.edu> Sent: Thursday, September 2, 2021 10:59 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) For those of you who have experienced this, what was your user load and how were your clusters operating (balancing, active/standby) ? I wonder if there’s a threshold.. Thx! From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Smith, Nayef Sent: Thursday, September 2, 2021 10:20 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) "I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving." To this point, the action we took that seemed to help the most was adjusting our active client load balancing threshold. We dropped it significantly to force clients to balance across controllers. Once we got below ~5000 active clients per controller, we stopped seeing the mass client connection issues. We still have a controller that hasn't taken significant load, but now that we've been running without major issues for the past few days, we're reluctant to touch the setting again. Nayef Z. Smith | Network Services | Voice: 404-727-6019 From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Davis, Jonathan Alan <jonath...@unc.edu> Sent: Thursday, September 2, 2021 9:27 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) Lee, don’t you bring your bad Cisco-juju to this conversation! :-) Now that Lee has been properly handled, this is probably a great opportunity to say ‘hello’ to the greater list. Hello! Last night, we (UNC) restarted the controller used to test the firewall policy. Despite Aruba’s advisory, we’ve been led to believe that restarting STM may not be enough, and restarting the whole controller may be required to resolve high STM CPU utilization. This morning we are keeping a close eye on that controller. While STM is surging well past 100%, it seems to be averaging much closer to 95%. However… We also only have about 7,000 users connected across the cluster. It will be interesting to see what happens as the day progresses and students wake up and migrate from the ResNET cluster to the Campus cluster. I’m also noticing that there are much fewer clients on this controller, and that ratio doesn’t seem to be improving. I’ll update as we progress through this. JD -- Jonathan Davis Wireless Architect The University of North Carolina at Chapel Hill jonath...@unc.edu +1 336 279 3355 (Mobile) From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Lee H Badman <lhbad...@syr.edu> Sent: Thursday, September 2, 2021 9:06:33 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing any issues in the fall with large classrooms and delayed connection times (Aruba 8.5.0.13) But you tested in your lab, right? I love that one… put new code on a couple of APs, or even a few dozen. That’s supposed to somehow indicate what will happen at bigger load… and also maybe implies the vendor didn’t do their own “similar lab testing”… “You should have tested before upgrading the whole environment…” how do you REALLY do that? And should you really have to? Just pondering the general state of things. > On Sep 2, 2021, at 08:59, Enfield, Chuck <cae...@psu.edu> wrote: > > That's been my experience for years. The network works great when there are > no students around. My working theory is that students emit RF interference, > but research ethics won’t let me run the tests, so we'll never know for sure. > > -----Original Message----- > From: The EDUCAUSE Wireless Issues Community Group Listserv > <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Patrick McEvilly > Sent: Thursday, September 2, 2021 8:56 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else seeing > any issues in the fall with large classrooms and delayed connection times > (Aruba 8.5.0.13) > > Speaking from experience, I would be very concerned. We had no issues until > students returned and we went downhill from there. > > > On 9/2/21, 8:50 AM, "The EDUCAUSE Wireless Issues Community Group Listserv on > behalf of Rob Harris" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU on behalf of > robert.har...@culinary.edu> wrote: > > Has anyone seen any details regarding what they consider "Large" > environments? We upgraded during the break, but both before and after > versions are affected. We didn't notice this happening before, should we be > concerned now? > > The "dropped" is 0 and the stm cpu usage is in single digits, but client > count is really low (they come back this weekend as well), could we be in the > clear? > > (asked the SE team and opened a tac call, same questions to them) > > thx > > -----Original Message----- > From: The EDUCAUSE Wireless Issues Community Group Listserv > <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Jason Healy > Sent: Thursday, September 2, 2021 8:45 AM > To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Anyone else > seeing any issues in the fall with large classrooms and delayed connection > times (Aruba 8.5.0.13) > > CAUTION: This email originated from outside The Culinary Institute of > America. Do not click links or open attachments unless you recognize the > sender and know the content is safe. > > FWIW, Aruba just posted an advisory regarding this issue: > > Aruba Support Advisory ARUBA-SA-20210901-PLVL04, "Wi-Fi Client > Connectivity Failures in Large Client Environments" > > Good luck to those of you hit by this. My students start coming back this > weekend so I'll be watching this closely! > > Jason > ********** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the message, > copy and paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3D&reserved=0 > > ********** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the message, > copy and paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3D&reserved=0 > > ********** > Replies to EDUCAUSE Community Group emails are sent to the entire community > list. If you want to reply only to the person who sent the message, copy and > paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=04%7C01%7Ccae104%40PSU.EDU%7C8d074518e4d44dbded4f08d96e110298%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637661841597428557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZlqC3lzdMWgYnKcohDgtGE4EVj%2BBAPD063ThuTr8sNU%3D&reserved=0 > > ********** > Replies to EDUCAUSE Community Group emails are sent to the entire community > list. If you want to reply only to the person who sent the message, copy and > paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
smime.p7s
Description: S/MIME cryptographic signature
