Brandon, We made following major changes over the summer:
1. changed F5 from 'bandwidth savings over WAN' load balancing to performance load balancing; 2. Use read-only LDAP servers; 3. moved radius servers to the same subnet that wireless controllers are in; 4. added a new 7240 wireless controller (now total 4 local controllers); 5. use anycast to pair up a wireless controller to a radius server. Change #1 improved backend from 20 auths/sec (login OKs) capable to over 170 auths/sec (login OKs) capable in my EAP authentication performance tests. Change #2 uses read-only LDAP servers as read-write servers tend to be busy as many apps and systems write updates to them (significantly reduced 'ldap_search() failed: LDAP connection lost' errors). Change #3 eliminates firewalls and L3 routing between controllers and radius servers. Change #5 clears out congestions between controller and radius server and give me a lot of flexibilities. I don't know how your system is configured. How many master and local controllers and how many radius and ldap servers in use and how did you configure them. Before this summer, we had three wireless controllers for campus, they were managed by a master controller. Since there is no load balancing on aruba controllers (I was told it's coming this fall), all controllers send auth requests to one radius server. This one radius server then interacts backend authentication servers. It became issues in peak times, normally 11am - 3pm Tue-Thur, when we saw radius server logs with 'Discarding duplicated requests...' errors. Could you briefly describe your wireless configurations (number of controllers, radius servers, ldap servers; eap settings; vlan assignments; do you also use captive portal; do your wireless controllers talk to ldap servers for captive portal and ssh/admin login authentication)? What errors do you see in radius logs? What's the highest auth/sec you can get during peak times (search and calculate from radius log or using graphing tools like cacti, prtg). Before summer, we got little over 20 auths/sec (login OKs) in peak time, now the highest we got was over 70 auths/sec. You mentioned you don't have F5, I believe anycast would be a good option for you. If you could drop me a few lines about your system, I would be happy to do my best to help. Yu Wang ____________________________ Network Architect Information Technology Services The Florida State University 850-645-6810 [email protected] -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Case, Brandon J Sent: Wednesday, August 27, 2014 3:22 PM To: [email protected] Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) Would you be able to elaborate on the improvements you did over the summer? We have a similar setup with regards to the backend, although ours is just freeradius -> ldap without the F5. Our usage levels are just a bit higher than yours but we're receiving lots of user reports of the inability to authenticate but nothing consistent enough to isolate and test repeatedly. Thanks, Brandon -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Wang, Yu Sent: Wednesday, August 27, 2014 3:15 PM To: [email protected] Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) Where are all your user accounts hosted? What kind of user database that serves the wireless system? Do you have a rough number of how many concurrent users at peak time? We had peak time wireless authentication failure issues in the past Spring semester. We did performance tests in the summer and found out it was the backend (F5 + LDAP). We did improvements in the summer and we have not seen the issue in the first three days of Fall semester. Yesterday's wireless usage set a new record with over 32k unique users and over 15k concurrent users. We use Aruba wireless with 802.1X, WPA2-Ent, PEAP, MSCHAPv2 + freeradius + F5 + ldap. It's different than yours but from the error you mentioned, it's likely the backend was congested. Yu Wang ____________________________ Network Architect Information Technology Services The Florida State University 850-645-6810 [email protected] -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Eric T. Barnett Sent: Wednesday, August 27, 2014 2:12 PM To: [email protected] Subject: [WIRELESS-LAN] Authentication failures at peak times (Cisco) We've got a relatively small deployment compared to many on this list, but we've run into a problem we just can't put our finger on. We're using 5508s and ISE as a RADIUS server and we're having HUGE latencies on WPA2-Enterprise PEAP authentication. There's times when almost no one can authenticate. What's really weird is that the controllers show "AAA Authentication Error" when this happens even though the username and password is correct. None of the devices seem distressed and there's no network problems we can see. Anyone ever seen this before or have any ideas how to troubleshoot? TAC so far has been not incredibly useful but they have only been on the case for a day or so now. I can hear my users sharpening the pitchforks... Thanks, Eric Barnett Wireless Administrator Information and Technology Services Arkansas State University 870 680 4243 ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
