Hi Dexter, Late last evening, Cisco's senior engineers said we could expect fix code by this Monday, and provided this input:
To give you a little background on this bug - this issue is mainly due race condition with rogue task and double call/release of a function causing crash. Unfortunately all this while until early this month we have not received the right dumps from other customers running into this bug hence we could not RCA. We have finally received full dump from a customer first week of March to root cause and fix this issue by March 5th. Again- Monday *should* bring relief, and I'm glad Cisco took yesterday's input to heart. -Lee Lee Badman Wireless/Network Architect ITS, Syracuse University 315.443.3003 (Blog: http://wirednot.wordpress.com) From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Dexter Caldwell Sent: Wednesday, March 18, 2015 8:42 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] WLC 5508 Reboots- 8.0.110.0 Code I definitely had that issue. Lee that bug search sounds exactly like what I was experiencing. At the bottom of this response is my 12/9/14 response on this list describing our experience. I eventually had to downgrade to 7.6.130 to get things stable. We have 6 controllers and I RMA'd 4 of them about 6 or 7 times because after the frequent reboots, they would occasionally die for good. (unable to boot or get output any on a console connection). Only power supplies worked when this happened. I suffered it for several weeks, but it was simply too risky because at some points I had 2-3 controllers offline at once in RMA status. (scary). The controllers would never log the crash event to the server so we could never record the reason. Sounds like they finally got this addressed, but only 8.0.115.0 is available from February, yet the bug report suggestions this was posted March 15th, so it seem unlikely to be fully fixed in 8.0.115.0 code. I guess 8.1(10.191) is a special request since it's not yet available for download- at least not for me. Dexter --------Previous Thread excerpt------------------ From: Dexter Caldwell Sent: Monday, December 8, 2014 5:57 PM To: 'dannyea...@rice.edu'; WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: RE: [WIRELESS-LAN] Cisco WLC 8.0.100.0 GO or No Go? I'm running the 8.0.100.0 code in production. I'd say for now it's a wait. I like a lot of things about it, but I've had two controllers completely die on it to where both were out of production at the same time! Full RMA and everything. I have a plan B, C and D so I was prepared to deal with it, but until we figure out what's causing our issue, (spontaneous reboots), it's hard to recommend it for production. More often it seems to happen when there is load but a few times not. It also does not seem to generate a crash log when it occurs so it's been evasive to troubleshoot. If you can spare to lose a controller or two of and still operate seamlessly, you'll may wan to consider it. You'll want to test for sure. There is no question everything is very fast in terms of recovery from restart. It is far faster than what it used to be for us on 7.3. I was willing to deal with the reboots since users barely if ever noticed, but the controllers dyeing was another matter entirely. Two of them rebooted fine one time and another time they rebooted and never recovered. When this occurs, we cannot get any console output. Replacement was the only route. If you're wondering, not all of these problems show up in test because there are significant load differences. May not be typical, and they're the only issues I've noticed, but they are not trivial. -------------------end excerpt--------------------- Dexter Caldwell Dir. Systems & Networks Information Technology Services Furman University 3300 Poinsett Hwy Greenville, SC 29613 email: dexter.caldw...@furman.edu<mailto:dexter.caldw...@furman.edu> office: 864-294-3566 facsimile: 864-294.3001 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jason Cook Sent: Wednesday, March 18, 2015 7:49 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] WLC 5508 Reboots- 8.0.110.0 Code Not sure we have hit that issue, but still seeing some crash issues since May last year We have had 5 PRD controllers on 110.0 since 1st Feb and another 1(dev) since December last year. One of the PRD crashed 2 days ago, but that is all we have seen. I'm not sure if ours is related to memory, it's suspiciously looks like a similar crash to what we have seen since 7.6.120 and every release in between. (yep, take that kick). Though we haven't seen anything in the logs(memory allocation failures) like we did the first rounds in 7.6.120.0 in the other crashes, the memory usage pattern has similarities. That is it will gradually rise from 55-70% (where it's normally stable) upto 80-85%, then crash. (see attached image). We rarely get crash logs or dumps for this. Has been less obvious and occurred less since 7.6.120.0 but still appears to show up. Hard to tell with the one 2 days ago, Prime graphs are currently doing weird things.... Surprise surprise. Another thing we found with the upgrade to 8.0 was that 2 controllers crashed while downloading the software, this was memory again. Both controllers exhibited that wonderful memory issue upon investigation.... So next time we'll check memory and do a pre-reboot... The upgrade worked the second time around. If you monitor cisco uptime on your devices you can get messages when a device reboots, handy for when the timing means you don't get the down message. There is a reset timer on them though, which can provide false alerts...we have never hit that on a controller, they aren't up long enough.. I think it's 2+years, so we have seen it on switches PROBLEM for 'cisco_uptime' on host 'c5508-01-net02-plazadc-nt' (129.127.xxx.xxx / Wireless Controller) 0d 0h 0m 10s ago (Wed Mar 18 06:54:09 ACDT 2015) ----------------------------------- Uptime CRITICAL - *26994* Performance Data: iso.3.6.1.2.1.1.3.0=26994 ----------------------------------- -- Jason Cook The University of Adelaide, AUSTRALIA 5005 Ph : +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Lee H Badman Sent: Thursday, 19 March 2015 12:23 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC 5508 Reboots- 8.0.110.0 Code Sigh... just kick me. Our latest Cisco WLAN fun comes in the form of 5508 spontaneous reboots on 8.0.110.0 code. Has anyone else on the list experienced this? I do find this Support Community thread: https://supportforums.cisco.com/discussion/12411926/wlc-5508-automatically-restarting-twice-week#comment-10362606 And this related bug: https://tools.cisco.com/bugsearch/bug/CSCuq74491 Have had one reboot today, and found that another had done so last week quick enough where monitoring and alerting didn't catch it. Now going through all of them to see if there might have been others missed. TAC case open and I see that 8.0.110.0 is no longer available to download, with 8.0.115.0 "recommended". -Lee Badman Lee Badman Wireless/Network Architect ITS, Syracuse University 315.443.3003 (Blog: http://wirednot.wordpress.com) ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
