Dan, What time is your controller showing? From your log messages it looks like its 2:16pm there? AP's have issues when the time is off between the controller and the ap's.
Craig Eyre Network Analyst IT Services Department Mount Royal University 4825 Mount Royal Gate SW Calgary AB T2P 3T5 P. 403.440.5199 E. [email protected] "The difference between a successful person and others is not a lack of strength, not a lack of knowledge, but rather in a lack of will." Vincent T. Lombardi From: Dan Brisson <[email protected]> To: [email protected] Date: 02/01/2012 09:32 AM Subject: Re: [WIRELESS-LAN] Cisco APs losing CAPWAP session Sent by: The EDUCAUSE Wireless Issues Constituent Group Listserv <[email protected]> This what we see on the AP at the time the AP disjoins: *Feb 1 14:16:25.174: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.246.207.214:5246 *Feb 1 14:16:25.227: %WIDS-5-DISABLED: IDS Signature is removed and disabled. *Feb 1 14:16:25.227: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY *Feb 1 14:16:25.227: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY *Feb 1 14:16:25.293: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down *Feb 1 14:16:25.293: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down *Feb 1 14:16:25.299: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Feb 1 14:16:25.309: status of voice_diag_test from WLC is false *Feb 1 14:16:25.309: %LINK-3-UPDOWN: Interface Dot11Radio1, changed state to up *Feb 1 14:16:25.318: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up -dan Dan Brisson Network Engineer University of Vermont (Ph) 802.656.8111 [email protected] On 2/1/2012 10:30 AM, Mike Goebel wrote: > Dan, have you tried logging into the AP itself and checking the logs > by chance? > > Mike > > On 2/1/2012 10:03 AM, Dan Brisson wrote: >> It does seem as though I've grabbed some folks attention. I sure hope it >> turns out to not be something simple. :) >> >> I could certainly try moving the APs around...easy enough to do, >> although from what we've seen, the pattern of AP drops is so totally >> random, hard to say if I'll see anything. At this point though, it's >> worth a shot. >> All interfaces clean and no QoS in place. >> >> Not sure if this will come through for everyone, but here's an example >> of what I see after an AP drops. This is from the controller, on the >> General tab for an AP: >> >> >> >> Thanks, >> -dan >> >> Dan Brisson >> Network Engineer >> University of Vermont >> (Ph) 802.656.8111 >> [email protected] >> >> >> On 2/1/2012 9:26 AM, Garry Peirce wrote: >>> I think you have some of us all getting curious! ;-) >>> >>> Could you put a historically stable admin AP onto the 5508 and >>> vice-versa to >>> see if behaviors change? >>> Do we assume that all switchports in the path are showing they're >>> running >>> clean? >>> Any QoS config in place on the switches? >>> >>> >>> -----Original Message----- >>> From: The EDUCAUSE Wireless Issues Constituent Group Listserv >>> [mailto:[email protected]] On Behalf Of Dan Brisson >>> Sent: Wednesday, February 01, 2012 9:09 AM >>> To:[email protected] >>> Subject: Re: [WIRELESS-LAN] Cisco APs losing CAPWAP session >>> >>> Good to know. >>> >>> The trunks are actually all 10Gig links, or 90% of them are, so >>> utilization >>> is most likely not the case, which I'm able to verify from Cacti >>> graphs. >>> The APs are connected to 3560Xs PoE switches that then uplink into >>> either a >>> 3560E-12D or directly into a 4900M where the 5508s are connected. >>> Certainly >>> can't rule out physical layer issue somewhere, although it's so wide >>> spread >>> across 2 different 5508s that we would need to have multiple issues. >>> The other interesting thing for us is that the 500 or so APs on our >>> admin >>> side that do not lose their CAPWAP session, join to WiSMs, not 5508s. >>> >>> Thanks, >>> -dan >>> >>> Dan Brisson >>> Network Engineer >>> University of Vermont >>> (Ph) 802.656.8111 >>> [email protected] >>> >>> >>> On 1/31/2012 8:44 PM, Garry Peirce wrote: >>>> We have ~1400 (1240s->3502's) running 7.0.116 and have no such issues. >>>> >>>> I would guess at packet loss as well - some things you might look at: >>>> Are the trunks carrying user/AP traffic seem congested when the APs >>>> drop? >>>> Have you verified there are no duplex issues? It may exhibit itself >>>> more as traffic levels rise. >>>> ResHall switching significantly different than on the admin side? >>>> >>>> Probably need further topology, version, config info, but as you've a >>>> case open, the TAC will likely ask the same and help find the >>>> culprit(s) >>> for you. >>>> >>>> -----Original Message----- >>>> From: The EDUCAUSE Wireless Issues Constituent Group Listserv >>>> [mailto:[email protected]] On Behalf Of Dan Brisson >>>> Sent: Tuesday, January 31, 2012 8:30 AM >>>> To:[email protected] >>>> Subject: [WIRELESS-LAN] Cisco APs losing CAPWAP session >>>> >>>> I'm curious if any Cisco users out there are experiencing or have >>>> experienced what we're seeing on our campus. This past summer we >>>> installed 3502i's in all of our residence halls - approximately 500 >>>> total. Ever since the students have moved in, we will get messages >>>> from WCS stating that "AP XYZ" is down and disassociated from the >>>> controller. When I check out the AP, the uptime is fine, but the >>>> "CAPWAP join time" is for like 30 seconds, or however long it took >>>> me to >>> check. >>>> We've tracked this and it is totally random as to what AP will drop, >>>> which makes troubleshooting this very tough. The log on the AP isn't >>> helpful. >>>> I'm working with TAC who suggests that keepalives are getting missed. >>>> I'm not sure why that would be the case since we have another >>>> 500 or so APs on the admin side that very rarely drop. Adding to >>>> that, when the students left for break, the AP drops stopped. They >>>> came back, and sure enough, the drops start up again. >>>> >>>> I will say that the AP always joins back immediately, but for the time >>>> that it does drop A) I'm sure connectivity is affected in that area >>>> and >>>> B) we get an email. >>>> >>>> Anyone experiencing this? >>>> >>>> Thanks, >>>> -dan >>>> >>>> >>>> -- >>>> Dan Brisson >>>> Network Engineer >>>> University of Vermont >>>> (Ph) 802.656.8111 >>>> [email protected] >>>> >>>> ********** >>>> Participation and subscription information for this EDUCAUSE >>>> Constituent Group discussion list can be found at >>> http://www.educause.edu/groups/. >>>> ********** >>>> Participation and subscription information for this EDUCAUSE >>>> Constituent >>> Group discussion list can be found athttp://www.educause.edu/groups/. >>> >>> ********** >>> Participation and subscription information for this EDUCAUSE >>> Constituent >>> Group discussion list can be found athttp://www.educause.edu/groups/. >>> >>> ********** >>> Participation and subscription information for this EDUCAUSE >>> Constituent Group discussion list can be found >>> athttp://www.educause.edu/groups/. >> ********** Participation and subscription information for this EDUCAUSE >> Constituent Group discussion list can be found at >> http://www.educause.edu/groups/. >> > ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
