How are you measuring the latency? Round trip from controller/ISE/AD? Or authentication latency in the System summary graph on the admin node?
Thanks, Brandon From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Joe Roth Sent: Tuesday, September 09, 2014 9:29 AM To: [email protected] Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) We are running client exclusion. Cisco has checked "all the usual stuff", and so far, nothing there. It seems to be somehow related to the replication between the nodes. When replication is not running between the policy nodes our authentications go through with low latency. To get replication to work we have to cut all authentication to the node, restart and resync it, then start sending authentications again, but the latency goes through the roof at that point. We haven't made any config changes to the SSID, and ISE has always handled this load in the past without a problem. On Mon, Sep 8, 2014 at 2:45 PM, Lee H Badman <[email protected]<mailto:[email protected]>> wrote: Are you running client exclusion on your 802.1X SSID? If not, it is worth looking at- I guarantee it. Can fill in more if interested. Been there, done that. -Lee From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Eric T. Barnett Sent: Monday, September 08, 2014 2:29 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) It was 1.2 fully patched. We haven’t upgraded to 1.2.1. That’s disheartening to hear though as we were hoping an upgrade would fix the problem. --Eric From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Joe Roth Sent: Thursday, September 04, 2014 9:39 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) Eric, Are you running 1.2 fully patched, or 1.2.1? We are seeing some serious issues right now with 1.2.1. We ended the Spring semester on 1.2 and things worked great, but our start up with 1.2.1 has gone terrible. We are seeing high radius/PEAP latency on our policy nodes. What is odd is that if we reboot a policy node and it comes back up but cannot synchronize, there is no latency at all, authentications go through. On Tue, Sep 2, 2014 at 5:14 PM, Eric T. Barnett <[email protected]<mailto:[email protected]>> wrote: You are right, that command can cause some serious problems. The good news is that we moved to a Microsoft RADIUS server as a temporary check. It works great! I still need to wait until tomorrow morning for full peak, but during medium load today it worked perfectly. It wasn’t working well even during this load with ISE. The bad news is apparently something is very wrong with our ISE installation. At least I’ve got my users off of my back for a bit while we figure out what’s wrong with ISE. --Eric From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Jeffrey Sessler Sent: Tuesday, September 02, 2014 9:46 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) I don't know if it was mentioned here, but you may want to look at this if you have defined more than on raidus server on your controlers: config radius aggressive-failover disable This turns off the aggressive failover of RADIUS - this prevents the situation where a unknown user attempts to connect, and the controllers consider the delay in response (or no response) as a failure. The controller will then switch to the other RADIUS server. This results in a ping-pong between radius servers. With the feature disabled, the controller only fails over to the next AAA server if there are three consecutive clients that fail to receive a response from the RADIUS server. Jeff >>> On Tuesday, September 02, 2014 at 5:21 AM, in message >>> <[email protected]<mailto:[email protected]>>, >>> "Case, Brandon J" <[email protected]<mailto:[email protected]>> wrote: Don, Yep the Timeout Requests counter on the controllers ticks up for the particular RADIUS server they’re talking to. I’ve also noticed the Pending Request timer increase at times but eventually it drops back to 0 when usage levels go down. Which vendor supported RADIUS appliances did you switch to? Thanks, Brandon From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Wright, Don Sent: Monday, September 01, 2014 9:17 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) Brandon, Can you see any radius issues based on stats on your controllers, timeouts, etc. We were seeing these on our FR servers last fall before we moved to our vendor support radius appliances. - Don Wright Lead Network Operations Engineer Brown University On Wed, Aug 27, 2014 at 3:21 PM, Case, Brandon J <[email protected]<mailto:[email protected]>> wrote: Would you be able to elaborate on the improvements you did over the summer? We have a similar setup with regards to the backend, although ours is just freeradius -> ldap without the F5. Our usage levels are just a bit higher than yours but we're receiving lots of user reports of the inability to authenticate but nothing consistent enough to isolate and test repeatedly. Thanks, Brandon -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Wang, Yu Sent: Wednesday, August 27, 2014 3:15 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) Where are all your user accounts hosted? What kind of user database that serves the wireless system? Do you have a rough number of how many concurrent users at peak time? We had peak time wireless authentication failure issues in the past Spring semester. We did performance tests in the summer and found out it was the backend (F5 + LDAP). We did improvements in the summer and we have not seen the issue in the first three days of Fall semester. Yesterday's wireless usage set a new record with over 32k unique users and over 15k concurrent users. We use Aruba wireless with 802.1X, WPA2-Ent, PEAP, MSCHAPv2 + freeradius + F5 + ldap. It's different than yours but from the error you mentioned, it's likely the backend was congested. Yu Wang ____________________________ Network Architect Information Technology Services The Florida State University 850-645-6810<tel:850-645-6810> [email protected]<mailto:[email protected]> -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Eric T. Barnett Sent: Wednesday, August 27, 2014 2:12 PM To: [email protected]<mailto:[email protected]> Subject: [WIRELESS-LAN] Authentication failures at peak times (Cisco) We've got a relatively small deployment compared to many on this list, but we've run into a problem we just can't put our finger on. We're using 5508s and ISE as a RADIUS server and we're having HUGE latencies on WPA2-Enterprise PEAP authentication. There's times when almost no one can authenticate. What's really weird is that the controllers show "AAA Authentication Error" when this happens even though the username and password is correct. None of the devices seem distressed and there's no network problems we can see. Anyone ever seen this before or have any ideas how to troubleshoot? TAC so far has been not incredibly useful but they have only been on the case for a day or so now. I can hear my users sharpening the pitchforks... Thanks, Eric Barnett Wireless Administrator Information and Technology Services Arkansas State University 870 680 4243<tel:870%20680%204243> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. -- Joe Roth Network Manager Binghamton University Ph. 607-777-7528<tel:607-777-7528> Fax 607-777-4009<tel:607-777-4009> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. -- Joe Roth Network Manager Binghamton University Ph. 607-777-7528 Fax 607-777-4009 ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
