One thing we did find is that by turning Multicast off (Controller-Multicast) it dropped the UDP traffic from 40-60Mb/sec down to 1-2Mb/sec on all Trunk Ports across campus. This was something even Cisco was surprised by, so maybe it’s something with the 7.0.116 code. ??? It was on by default after the upgrade because I don’t remember ever enabling it since we don’t use Multicast over wireless, just on the wired network.
All our AP’s and controllers are on the same Vlan, so we’ve ruled out the router/firewall, and none of the Gig trunk ports are even near capacity. We are starting to make progress, but the biggest thing we’re seeing now is the massive interference which we’re working on. Thanks Shayne From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jeffrey Sessler Sent: Thursday, October 27, 2011 7:44 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms Getting rid of the rogues can't hurt, but this smells like a network issue. The communication between the AP and controller are such that it wouldn't take much to cause the AP's to see a problem and try to fix it. Even an etherchannel flapping, say on a trunk heading to those buildings, would be enough to cause the APs to go back into CAPWAP discovery. Oh, and if you've not yet escalated this to your Cisco team, you should. Once the wireless business unit is involved, they tend to resolve problems nearly at the speed of light... well... except if it involves Lee. ;) Jeff >>> On Thursday, October 27, 2011 at 9:53 AM, in message >>> <700d77bb392ae543b5b4455c8db89e3a09cc7...@mbox1.ad.bradley.edu>, "Ghere, >>> Shayne" <sgh...@bumail.bradley.edu> wrote: Thank you for the input. This is how we have the ports setup currently. We’re to the point of experimenting with certain buildings in the dorms and turning all “rogue” wireless devices off (including wireless printers) to see if that helps. I’ll let you know what we find, and hopefully this will be resolved before I retire in 22 years. ;) Thanks again everyone! Shayne From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Wednesday, October 26, 2011 5:05 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms Ditto that. Lee H. Badman Wireless/Network Engineer, ITS Adjunct Instructor, iSchool Syracuse University 315.443.3003 ________________________________ From: The EDUCAUSE Wireless Issues Constituent Group Listserv [WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] on behalf of Jeffrey Sessler [j...@scrippscollege.edu] Sent: Wednesday, October 26, 2011 5:48 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms As Lee mentions, the communication between the AP and the controller is via a standard access port. There should be no need to have trunking or and other configuration on the port for the AP. Even if the AP has multiple SSIDs and VLANs, all of that traffic is encapsulated within CAPWAP. Every port an AP connects to on my campus looks something like this: interface GigabitEthernet1/0/1 description Cisco AP switchport access vlan 111 switchport mode access spanning-tree portfast >>> On Wednesday, October 26, 2011 at 7:02 AM, in message >>> <700d77bb392ae543b5b4455c8db89e3a09cc7...@mbox1.ad.bradley.edu>, "Ghere, >>> Shayne" <sgh...@bumail.bradley.edu> wrote: Lee, I've read multiple documents and all say different things on setup. We have an internal registration system that we register each AP's mac address and it's updated (yes we're still using VMPS) in the vmps.cfg file. So currently we have each port setup like this: interface GigabitEthernet0/48 description GPB-AIR2-2 2-16 switchport access vlan dynamic no logging event link-status no snmp trap link-status spanning-tree portfast We have turned off Spanning-tree, hardcoded it to VLAN 5, hard set it to 1000/Full and even had them setup as trunks to allow ONLY the vlans we are passing for wireless, but to no avail. If you have any ideas, please let me know. Thanks Shayne P.S. Sorry, our e-mail was out yesterday so I couldn't log in to read or respond. -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Monday, October 24, 2011 6:08 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms Shayne, please post what your switchport configs look like for the APs. Also, are you managing the APs on a single network? Lee H. Badman Wireless/Network Engineer, ITS Adjunct Instructor, iSchool Syracuse University 315.443.3003 ________________________________________ From: The EDUCAUSE Wireless Issues Constituent Group Listserv [WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] on behalf of Jeffrey Sessler [j...@scrippscollege.edu] Sent: Monday, October 24, 2011 6:10 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms This sounds/looks a lot more like a network issue then an AP/rogue problem. The logs suggest the AP's are having problems staying in contact with the controllers. Everything gigabit from end to end? What does WCS indicate as the number of channel changes per hour? As a test, on each 5508, under Wireless, 802.11b/g/n and 802.11a/n, DCA, change the interval to 6 hours, and the DCA Channel Sensitivity to low. Do this - contact your Cisco team and ask them to put you in contact with the Wireless Business unit. They have a team that can assist if you feel it's a show-stopper problem. Jeff >>> On Monday, October 24, 2011 at 7:30 AM, in message <700d77bb392ae543b5b4455c8db89e3a09cc7...@mbox1.ad.bradley.edu>, "Ghere, Shayne" <sgh...@bumail.bradley.edu> wrote: Hello Craig, The upgrades were done independently as was the WCS upgrade. I have the WLC's setup with an equal number of AP's on each with Primary, Secondary and Tertiary controllers configured so I can upgrade one with them fail over to the others. I have spent the past 3 weeks pouring over the configurations on each WLC...even doing print screens of each to compare and they are completely identical. The WCS (running 7.0.172.0) has settings that the WLC's don't like (7.0.116.0) or are mismatched such as power level settings etc. So for those I just set on the WLC's and don't push out the templates on the WCS with the odd values. Here's our history: May 2010 we started with the 3 WS-C5508 and 1 WCS to handle all the dorms/academic/non academic buildings. We are a single campus (so no wan). We were running 6.0.199.0 on the WLC's (I believe) and there were a few problems. Cisco advised us to upgrade to 6.0.299.0 (maintenance release) to fix another issue, then 4 weeks ago upgraded to 7.0.116.0 to fix another problem. The WCS needed to be upgraded to 7.0.172.0 since the code we were running before wouldn't work with the 116.0 on the WLC's. /ugh Since the students moved back to campus we've had nothing but radios shutting off, AP's completely rebooting, AP's moving to secondary/tertiary controllers then back to the primary etc. It has been a NIGHTMARE. What I'm seeing is that our B/G channels 1,6,11 are also being used by the 450+ rogue AP's in the dorms. We can't shut off the B/G due to older machines, but the interference causes the A/N radios to drop as well. When we were running Autonomous AP's we didn't have this problem, but since moving to LWAPP we've had problems. Thanks Shayne From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Craig Eyre Sent: Monday, October 24, 2011 9:02 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Problems in the Dorms Shayne, Nothing jumps out at me, but I do have a couple questions. 1. What version were you running previously? 2. Did you deploy the upgrade with your WCS? The reason I ask about the where you upgraded it from, is because a colleague I know just upgraded 2 WLC's from his WCS and the settings were different for each wlc. Regards, Craig Eyre Network Analyst IT Services Department Mount Royal University 4825 Mount Royal Gate SW Calgary AB T2P 3T5 P. 403.440.5199 E. ce...@mtroyal.ca "The difference between a successful person and others is not a lack of strength, not a lack of knowledge, but rather in a lack of will." Vincent T. Lombardi From: "Ghere, Shayne" <sgh...@bumail.bradley.edu> To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Date: 10/22/2011 04:53 PM Subject: [WIRELESS-LAN] Problems in the Dorms Sent by: The EDUCAUSE Wireless Issues Constituent Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> ________________________________ Hello, We currently provide wireless for all our Dorms using Cisco 1142N AP's, 1 WCS and 3 WLC5508's. We have roughly 375 AP's in the dorms but more than 450 rogue AP's that the students brought with them. Since we have no policy to disallow them bringing their own devices, we now have a mess. What we're seeing are the AP's either completely rebooting, radios shutting down then coming back up, or if the students are able to connect they get dropped after a few minutes. On the Academic side of the University we don't see this problem, however all the AP's are disassociating with the controllers every hour, then reassociating again. The WLC's are running 7.0.116.0 and the WCS is running 7.0.172.0. It appears that since upgrading the controllers to 7.0.116.0 the problems started with the disassociating/reassociating with no explanation. We are using WS-C2960S-PoE switches fibered to the core (6509) and have spent almost 28 hours on the phone with Cisco Tac looking at logs/packet captures and configuration review. Nothing is misconfigured and the packet captures show the following from one of the AP's: Oct 19 20:55:54.918: %CAPWAP-3-EVENTLOG: Retransmission Count= 3 Max Re-Transmission Value=3 *Oct 19 20:55:54.918: %CAPWAP-3-EVENTLOG: Max retransmission count exceeded going back to DISCOVER mode. *Oct 19 20:55:54.918: %CAPWAP-3-EVENTLOG: The function which Posted the message to send out of the box is wtpSendEchoReques and of Type=1 ., 1)19 20:55:54.918: %CAPWAP-3-EVENTLOG: Retransmission count for packet exceeded max(CAPWAP_ECHO_REQUEST *Oct 19 20:55:54.918: %CAPWAP-3-EVENTLOG: GOING BACK TO DISCOVER MODE *Oct 19 20:55:54.962: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 136.176.x.x:5246 *Oct 19 20:55:54.962: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Teardown. *Oct 19 20:55:54.963: %CAPWAP-3-EVENTLOG: DTLS session cleanup completed. Restarting capwap state machine. *Oct 19 20:55:55.006: %WIDS-5-DISABLED: IDS Signature is removed and disabled. *Oct 19 20:55:55.008: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY *Oct 19 20:55:55.008: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY *Oct 19 20:55:55.063: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down *Oct 19 20:55:55.063: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down *Oct 19 20:55:55.065: %CAPWAP-3-EVENTLOG: CAPWAP state not up. Abort sending channel and power levels info.136:176:x.x *Oct 19 20:55:55.074: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset *Oct 19 20:55:55.075: %CAPWAP-3-EVENTLOG: CAPWAP state not up. Abort sending channel and power levels info.136:176:x.x We're completely at a loss since none of the switch ports, trunk ports or the WLC's are showing dropped packets. Has anyone run into this problem and found a work around? I would greatly appreciate any help in this matter! Thanks Shayne ----------------------------- Bradley University T. Shayne Ghere, CCNA Network Engineer 1501 W. Bradley Ave. Morgan Hall, Suite 205 Peoria, IL 61625 sgh...@bradley.edu (309) 677-3094 ofc. (309) 677-3460 fax Class 2011 FBI CA Graduate ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. <http://www.educause.edu/groups/> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. <http://www.educause.edu/groups/> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. <http://www.educause.edu/groups/> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. <http://www.educause.edu/groups/> ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
