Re: [WIRELESS-LAN] strange WLC behavior

[Matthew Newton]

On Thu, Dec 03, 2015 at 04:17:12PM +, Oliver Elliott wrote:
> The 7.6.x range was buggy as hell so I'm not surprised. Get off there asap!

Not as buggy as 7.4.x... we ran 7.6 for a year quite happily.

All Cisco software releases are buggy... just depends on whether
the bugs affect your particular environment :)

> On 3 December 2015 at 16:15, John York  wrote:
> 
> > After a year of pretty much rock solid behavior we’ve had two instances
> > this week where EAP failed for some or all of the users on our WLC 5508

In what way?

> > experiencing the problem, but the WebAuth SSID worked fine.  The ACS logs
> > showed “EAP session timed out.”  The Windows NPS logs didn’t show any
> > authentication failures.

How many authentications per second? Is it busier than usual?

Could be a case of the WLC reusing RADIUS session IDs which will
totally break stuff and is a know issue under high numbers of
authentications.

Cisco have gone some way to fix this issue in the latest 8.x, but
as far as I'm concerned their RADIUS client design is overall
still pretty bad.

> > After a few hours it fixed itself.  I tried a 5508 reboot in one of the
> > instances, and it didn’t appear to help.

So likely behaviour caused by some external factor, such as the
above. But could be anything like eap timers not tuned well,
wireless issues at the edge, etc. Or backend auth being slow.

Cheers,

Matthew


-- 
Matthew Newton, Ph.D. 

Systems Specialist, Infrastructure Services,
I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom

For IT help contact helpdesk extn. 2253, 

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.

RE: [WIRELESS-LAN] strange WLC behavior

[Danny Eaton]

All Cisco software releases are buggy... just depends on whether the bugs 
affect your particular environment :)

Amen to that, and will say "All software is buggy".  

We're running 8.0.110.11 now for the past year or so, with no ill effects; with 
WiSM-2 HA clusters.  

-Original Message-
From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Matthew Newton
Sent: Thursday, December 03, 2015 10:37 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] strange WLC behavior

On Thu, Dec 03, 2015 at 04:17:12PM +, Oliver Elliott wrote:
> The 7.6.x range was buggy as hell so I'm not surprised. Get off there asap!

Not as buggy as 7.4.x... we ran 7.6 for a year quite happily.

All Cisco software releases are buggy... just depends on whether the bugs 
affect your particular environment :)

> On 3 December 2015 at 16:15, John York <yo...@brcc.edu> wrote:
> 
> > After a year of pretty much rock solid behavior we’ve had two 
> > instances this week where EAP failed for some or all of the users on 
> > our WLC 5508

In what way?

> > experiencing the problem, but the WebAuth SSID worked fine.  The ACS 
> > logs showed “EAP session timed out.”  The Windows NPS logs didn’t 
> > show any authentication failures.

How many authentications per second? Is it busier than usual?

Could be a case of the WLC reusing RADIUS session IDs which will totally break 
stuff and is a know issue under high numbers of authentications.

Cisco have gone some way to fix this issue in the latest 8.x, but as far as I'm 
concerned their RADIUS client design is overall still pretty bad.

> > After a few hours it fixed itself.  I tried a 5508 reboot in one of 
> > the instances, and it didn’t appear to help.

So likely behaviour caused by some external factor, such as the above. But 
could be anything like eap timers not tuned well, wireless issues at the edge, 
etc. Or backend auth being slow.

Cheers,

Matthew


--
Matthew Newton, Ph.D. <m...@le.ac.uk>

Systems Specialist, Infrastructure Services, I.T. Services, University of 
Leicester, Leicester LE1 7RH, United Kingdom

For IT help contact helpdesk extn. 2253, <ith...@le.ac.uk>

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.

!DSPAM:911,56606fe0195231016456774!

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.

RE: [WIRELESS-LAN] strange WLC behavior

[T. Shayne Ghere]

We moved off that as soon as the 8.0.120.x was out.  Make sure your AP's can
support 8.x code before you migrate to it.  95% of the issues we had on 7.4
and 7.6 went away once we moved to the new software.

If you're not running LAG, that will create problems in the 7.x software.

S

-Original Message-
From: The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Matthew Newton
Sent: Thursday, December 03, 2015 10:37 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] strange WLC behavior

On Thu, Dec 03, 2015 at 04:17:12PM +, Oliver Elliott wrote:
> The 7.6.x range was buggy as hell so I'm not surprised. Get off there
> asap!

Not as buggy as 7.4.x... we ran 7.6 for a year quite happily.

All Cisco software releases are buggy... just depends on whether the bugs
affect your particular environment :)

> On 3 December 2015 at 16:15, John York <yo...@brcc.edu> wrote:
>
> > After a year of pretty much rock solid behavior we’ve had two
> > instances this week where EAP failed for some or all of the users on
> > our WLC 5508

In what way?

> > experiencing the problem, but the WebAuth SSID worked fine.  The ACS
> > logs showed “EAP session timed out.”  The Windows NPS logs didn’t
> > show any authentication failures.

How many authentications per second? Is it busier than usual?

Could be a case of the WLC reusing RADIUS session IDs which will totally
break stuff and is a know issue under high numbers of authentications.

Cisco have gone some way to fix this issue in the latest 8.x, but as far as
I'm concerned their RADIUS client design is overall still pretty bad.

> > After a few hours it fixed itself.  I tried a 5508 reboot in one of
> > the instances, and it didn’t appear to help.

So likely behaviour caused by some external factor, such as the above. But
could be anything like eap timers not tuned well, wireless issues at the
edge, etc. Or backend auth being slow.

Cheers,

Matthew


--
Matthew Newton, Ph.D. <m...@le.ac.uk>

Systems Specialist, Infrastructure Services, I.T. Services, University of
Leicester, Leicester LE1 7RH, United Kingdom

For IT help contact helpdesk extn. 2253, <ith...@le.ac.uk>

**
Participation and subscription information for this EDUCAUSE Constituent
Group discussion list can be found at http://www.educause.edu/groups/.

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.

Re: [WIRELESS-LAN] strange WLC behavior

[Oliver Elliott]

The 7.6.x range was buggy as hell so I'm not surprised. Get off there asap!

On 3 December 2015 at 16:15, John York  wrote:

> After a year of pretty much rock solid behavior we’ve had two instances
> this week where EAP failed for some or all of the users on our WLC 5508
> (7.6.130.0).  For some users it uses EAP-PEAP-MSChapV2 to a Windows AD
> server running NPS.  For others it uses EAP-TLS to Cisco ACS.  Both were
> experiencing the problem, but the WebAuth SSID worked fine.  The ACS logs
> showed “EAP session timed out.”  The Windows NPS logs didn’t show any
> authentication failures.
>
>
>
> After a few hours it fixed itself.  I tried a 5508 reboot in one of the
> instances, and it didn’t appear to help.  None of the certs involved have
> expired and there haven’t been any recent configuration changes.
>
> I was going to upgrade to one of Cisco’s suggested WLC software versions
> over Christmas break—maybe this weekend would be better.
>
>
>
> Thanks
>
> John
> ** Participation and subscription information for this EDUCAUSE
> Constituent Group discussion list can be found at
> http://www.educause.edu/groups/.
>
>


-- 
Oliver Elliott
Senior Network Specialist
IT Services
University of Bristol
e: oliver.elli...@bristol.ac.uk
t: 0117 39 (41131)

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.

Re: [WIRELESS-LAN] strange WLC behavior

[John York]

>> > After a year of pretty much rock solid behavior we’ve had two instances
>> > this week where EAP failed for some or all of the users on our WLC 5508
>
>In what way?

Clients just wouldn't connect.  I didn't find anything in the WLC logs that 
helped me, but probably I just didn't understand what I was seeing.  I did see 
one iPad that made the user accept the cert for our CA, Entrust, but that's 
about it.

>
>> > experiencing the problem, but the WebAuth SSID worked fine.  The ACS logs
>> > showed “EAP session timed out.”  The Windows NPS logs didn’t show any
>> > authentication failures.
>
>How many authentications per second? Is it busier than usual?
>

We're tiny, only 65 APs, currently about 300 users on EAP SSIDs and max 1500 
authentications per hour.  Let's see, 1500/3600 is about 0.4 ;-)  This started 
sometime overnight, and our peak period is lunchtime.

>Could be a case of the WLC reusing RADIUS session IDs which will
>totally break stuff and is a know issue under high numbers of
>authentications.
>
>Cisco have gone some way to fix this issue in the latest 8.x, but
>as far as I'm concerned their RADIUS client design is overall
>still pretty bad.
>
>> > After a few hours it fixed itself.  I tried a 5508 reboot in one of the
>> > instances, and it didn’t appear to help.
>
>So likely behaviour caused by some external factor, such as the
>above. But could be anything like eap timers not tuned well,
>wireless issues at the edge, etc. Or backend auth being slow.
>
>Cheers,
>
>Matthew

I'll try going to 8.0.121.0 this weekend since that's easy, and falling back is 
easy (usually, knock on wood.)

Thanks everyone!
John

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/groups/.