Eric, Something that TAC had us do today that seems to have helped is to disable endpoint protection service on the PAN then reboot a PSN. I have no idea why this has helped. It's under administration-->settings-->endpoint protection service.
We have rebooted one PSN and it came back, sync'd and looks good now. We will try the others tomorrow a.m. and see if we have success with those. A couple of other things that we have run into in the past - are you using profiling? If so, try disabling it. This can be VERY resource intensive. It shouldn't be in 1.2, but you never know. Are you running this in VM, or appliances? If it's VM, take a look at the resources allocated to the servers. You may need to add CPUs/memory. Are VM tools up to date? On Tue, Sep 9, 2014 at 3:53 PM, Lee H Badman <[email protected]> wrote: > Are you seeing lots of failed auths in the ISE? Like inordinate amounts > from misconfigured/outside client devices? > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto: > [email protected]] *On Behalf Of *Eric T. Barnett > *Sent:* Tuesday, September 09, 2014 2:53 PM > > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > Actually, I had not been doing that previously, but I do now. It didn’t > have an appreciable effect. > > > > --Eric > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [ > mailto:[email protected] > <[email protected]>] *On Behalf Of *Lee H Badman > *Sent:* Monday, September 08, 2014 1:46 PM > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > Are you running client exclusion on your 802.1X SSID? If not, it is worth > looking at- I guarantee it. Can fill in more if interested. > > > > Been there, done that. > > > > -Lee > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [ > mailto:[email protected] > <[email protected]>] *On Behalf Of *Eric T. Barnett > *Sent:* Monday, September 08, 2014 2:29 PM > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > It was 1.2 fully patched. We haven’t upgraded to 1.2.1. That’s > disheartening to hear though as we were hoping an upgrade would fix the > problem. > > > > --Eric > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [ > mailto:[email protected] > <[email protected]>] *On Behalf Of *Joe Roth > *Sent:* Thursday, September 04, 2014 9:39 AM > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > Eric, > > Are you running 1.2 fully patched, or 1.2.1? We are seeing some serious > issues right now with 1.2.1. We ended the Spring semester on 1.2 and things > worked great, but our start up with 1.2.1 has gone terrible. We are seeing > high radius/PEAP latency on our policy nodes. What is odd is that if we > reboot a policy node and it comes back up but cannot synchronize, there is > no latency at all, authentications go through. > > > > On Tue, Sep 2, 2014 at 5:14 PM, Eric T. Barnett <[email protected]> > wrote: > > You are right, that command can cause some serious problems. > > > > The good news is that we moved to a Microsoft RADIUS server as a temporary > check. It works great! I still need to wait until tomorrow morning for full > peak, but during medium load today it worked perfectly. It wasn’t working > well even during this load with ISE. The bad news is apparently something > is very wrong with our ISE installation. At least I’ve got my users off of > my back for a bit while we figure out what’s wrong with ISE. > > > > --Eric > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto: > [email protected]] *On Behalf Of *Jeffrey Sessler > *Sent:* Tuesday, September 02, 2014 9:46 AM > > > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > I don't know if it was mentioned here, but you may want to look at this if > you have defined more than on raidus server on your controlers: > > > > *config radius aggressive-failover disable* > > > > This turns off the aggressive failover of RADIUS - this prevents the > situation where a unknown user attempts to connect, and the controllers > consider the delay in response (or no response) as a failure. The > controller will then switch to the other RADIUS server. This results in a > ping-pong between radius servers. With the feature disabled, the controller > only fails over to the next AAA server if there are three consecutive > clients that fail to receive a response from the RADIUS server. > > > > Jeff > > > >>> On Tuesday, September 02, 2014 at 5:21 AM, in message < > [email protected]>, "Case, > Brandon J" <[email protected]> wrote: > > Don, > > > > Yep the Timeout Requests counter on the controllers ticks up for the > particular RADIUS server they’re talking to. I’ve also noticed the Pending > Request timer increase at times but eventually it drops back to 0 when > usage levels go down. Which vendor supported RADIUS appliances did you > switch to? > > > > Thanks, > > Brandon > > > > *From:* The EDUCAUSE Wireless Issues Constituent Group Listserv [ > mailto:[email protected] > <[email protected]>] *On Behalf Of *Wright, Don > *Sent:* Monday, September 01, 2014 9:17 PM > *To:* [email protected] > *Subject:* Re: [WIRELESS-LAN] Authentication failures at peak times > (Cisco) > > > > Brandon, > > Can you see any radius issues based on stats on your controllers, > timeouts, etc. We were seeing these on our FR servers last fall before we > moved to our vendor support radius appliances. > > - > > Don Wright > > Lead Network Operations Engineer > > Brown University > > > > > > On Wed, Aug 27, 2014 at 3:21 PM, Case, Brandon J <[email protected]> wrote: > > Would you be able to elaborate on the improvements you did over the > summer? We have a similar setup with regards to the backend, although ours > is just freeradius -> ldap without the F5. Our usage levels are just a bit > higher than yours but we're receiving lots of user reports of the inability > to authenticate but nothing consistent enough to isolate and test > repeatedly. > > Thanks, > Brandon > > > -----Original Message----- > From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto: > [email protected]] On Behalf Of Wang, Yu > Sent: Wednesday, August 27, 2014 3:15 PM > To: [email protected] > > Subject: Re: [WIRELESS-LAN] Authentication failures at peak times (Cisco) > > Where are all your user accounts hosted? What kind of user database that > serves the wireless system? Do you have a rough number of how many > concurrent users at peak time? > > We had peak time wireless authentication failure issues in the past Spring > semester. We did performance tests in the summer and found out it was the > backend (F5 + LDAP). We did improvements in the summer and we have not seen > the issue in the first three days of Fall semester. Yesterday's wireless > usage set a new record with over 32k unique users and over 15k concurrent > users. > > We use Aruba wireless with 802.1X, WPA2-Ent, PEAP, MSCHAPv2 + freeradius + > F5 + ldap. It's different than yours but from the error you mentioned, it's > likely the backend was congested. > > > > Yu Wang > ____________________________ > Network Architect > Information Technology Services > The Florida State University > 850-645-6810 > [email protected] > > > -----Original Message----- > From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto: > [email protected]] On Behalf Of Eric T. Barnett > Sent: Wednesday, August 27, 2014 2:12 PM > To: [email protected] > Subject: [WIRELESS-LAN] Authentication failures at peak times (Cisco) > > We've got a relatively small deployment compared to many on this list, but > we've run into a problem we just can't put our finger on. We're using 5508s > and ISE as a RADIUS server and we're having HUGE latencies on > WPA2-Enterprise PEAP authentication. There's times when almost no one can > authenticate. What's really weird is that the controllers show "AAA > Authentication Error" when this happens even though the username and > password is correct. None of the devices seem distressed and there's no > network problems we can see. Anyone ever seen this before or have any ideas > how to troubleshoot? TAC so far has been not incredibly useful but they > have only been on the case for a day or so now. I can hear my users > sharpening the pitchforks... > > Thanks, > > Eric Barnett > Wireless Administrator > Information and Technology Services > Arkansas State University > 870 680 4243 > > ********** > Participation and subscription information for this EDUCAUSE Constituent > Group discussion list can be found at http://www.educause.edu/groups/. > > ********** > Participation and subscription information for this EDUCAUSE Constituent > Group discussion list can be found at http://www.educause.edu/groups/. > > ********** > Participation and subscription information for this EDUCAUSE Constituent > Group discussion list can be found at http://www.educause.edu/groups/. > > > > ********** Participation and subscription information for this EDUCAUSE > Constituent Group discussion list can be found at > http://www.educause.edu/groups/. > > ********** Participation and subscription information for this EDUCAUSE > Constituent Group discussion list can be found at > http://www.educause.edu/groups/. > > > > > -- > > Joe Roth > Network Manager > Binghamton University > Ph. 607-777-7528 > Fax 607-777-4009 > > ********** Participation and subscription information for this EDUCAUSE > Constituent Group discussion list can be found at > http://www.educause.edu/groups/. > -- Joe Roth Network Manager Binghamton University Ph. 607-777-7528 Fax 607-777-4009 ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
