It was indeed a space issue. I cleared out a couple directories, which put the free space on flash: at around 17MB and retried the “download backup”, which was successful.
I know for this AP I had to do a bios level recovery, which entailed loading a different firmware than what was on the AP. I’m sure this contributed the lack of space, but regardless, if for whatever reason space is tight on the AP you definitely want to manually clean house. -dan Dan Brisson Network Engineer University of Vermont From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Daniel Brisson Sent: Tuesday, March 21, 2017 1:38 PM To: [email protected] Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Googling a bit has yielded the possible explanation that there may not be enough space in flash. I’m trying to see if I can free up any. -dan Dan Brisson Network Engineer University of Vermont From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Daniel Brisson Sent: Tuesday, March 21, 2017 1:35 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Jeff, I have an AP at my desk that is failing to install the “backup” image….here’s the console output: lwapp_predownload_upgrade_proc: encounter flash problem, retry here examining image... extracting info (328 bytes) Image info: Version Suffix: k9w8-.v153_3_jc.201611091800 Image Name: ap3g2-k9w8-mx.v153_3_jc.201611091800 Version Directory: ap3g2-k9w8-mx.v153_3_jc.201611091800 Ios Image Size: 11622912 Total Image Size: 13773312 Image Feature: WIRELESS LAN|LWAPP Image Family: AP3G2 Wireless Switch Management Version: 8.2.131.40 MwarVersion:08028328.First AP Supported Version:07010100. Image version check passed Extracting files... ap3g2-k9w8-mx.v153_3_jc.201611091800/ (directory) 0 (bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/E5.bin (2213 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/uart_firmware_upgrade.bin (18239 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/ap3g2-k9w8-tx.v153_3_jc.201611091800 (73 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/MCU.bin (8799 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/info (328 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/ap3g2-bl-3600 (189183 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/X5.bin (1916 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/R5.bin (4547 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/V5.bin (514 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/HA2.bin (5840 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/F2.bin (15184 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/E2.bin (19856 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/ap3g2-bl-2600 (190140 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/img_sign_rel.cert (1375 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/C2.bin (21024 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/triggerfish.jed (0 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/C5.bin (15893 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/final_hash.sig (513 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/Q2.bin (8176 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/ap3g2-k9w8-mx.v153_3_jc.201611091800 (215867 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/8006.img (571726 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/X2.bin (16352 bytes) ap3g2-k9w8-mx.v153_3_jc.201611091800/html/ (directory) 0 (bytes) ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/ (directory) 0 (bytes) ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/ (directory) 0 (bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapSummary.htm (985 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapEvent.shtml.gz (988 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapConfig.shtml.gz (2864 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapBanner.htm (7514 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapMain.shtml.gz (3350 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/15/officeExtendapHelp.htm (5721 bytes) ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/ (directory) 0 (bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/sitewide.js (17250 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/ap_home.shtml.gz (1370 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/officeExtendap.css (41801 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/config.js (29225 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/config-oeap.js (779 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/appsui.js (563 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/back.shtml (512 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/forms.js (20442 bytes) ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/ (directory) 0 (bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/info.gif (399 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/background_web41.jpg (732 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/cisco-logo-2007.gif (1648 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/login_homeap.gif (19671 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/html/level/1/images/itp-logo.png (2822 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/Y2.bin (7008 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/HA5.bin (2049 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/V2.bin (12848 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/U5.bin (3609 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/8004.img (569124 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/Q5.bin (3957 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/img_sign_rel_sha2.cert (1371 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/F5.bin (4220 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/file_hashes (8007 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/R2.bin (15184 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/B5.bin (2333 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/Y5.bin (1875 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/U2.bin (8176 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/B2.bin (10512 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/final_hash (141 bytes) extracting ap3g2-k9w8-mx.v153_3_jc.201611091800/ap3g2-k9w8-xx.v153_3_jc.201611091800 (11615866 bytes) extracting info.ver (328 bytes) *Mar 21 00:47:11.666: Currently running a Release Image validate_sha2_block: Failed to get certificate chain *Mar 21 00:47:11.690: Using SHA-1 signed certificate for image signing validation. *Mar 21 00:47:11.834: Image signing certificate validation succeeded. ERROR: Problem installing (renaming) downloaded files *Mar 21 00:47:18.858: AP image integrity check PASSED -dan Dan Brisson Network Engineer University of Vermont (Ph) 802.656.8111 [email protected]<mailto:[email protected]> From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Jeffrey D. Sessler Sent: Tuesday, March 21, 2017 12:55 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Lee, Would you send me the output of a show-tech from that AP? Jeff From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> on behalf of "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, March 20, 2017 at 9:52 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Yes it does- every time. Lee Badman | Network Architect Adjunct Instructor | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e [email protected]<mailto:[email protected]> w its.syr.edu SYRACUSE UNIVERSITY syr.edu From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Jeffrey D. Sessler Sent: Monday, March 20, 2017 10:37 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Lee, reboot that AP, then use the “download backup” in the WLC GUI. Does it still fail? Jeff From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> on behalf of "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, March 20, 2017 at 6:52 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Not sure what I’m thinking, to be honest, Mike. Here’s the file system from a healthy 3702 AP, where primary code is 8.2.141.0 and backup is 8.2.121.0: haven422b-9088#dir flash: Directory of flash:/ 2 -rwx 287 Jan 1 1970 00:05:23 +00:00 info 3 -rwx 75 Mar 20 2017 13:35:56 +00:00 capwap-saved-config 38 drwx 576 Mar 1 1993 00:05:19 +00:00 ap3g2-rcvk9w8-mx 4 -rwx 65517 Dec 20 2016 09:47:23 +00:00 event.log 5 -rwx 0 Mar 1 1993 00:00:34 +00:00 config.txt 68 -rwx 266 Dec 20 2016 10:26:25 +00:00 env_vars 8 -rwx 12312 Mar 20 2017 13:36:48 +00:00 private-multiple-fs 80 drwx 0 Mar 1 1993 00:01:04 +00:00 configs 81 -rwx 75 Mar 20 2017 13:36:48 +00:00 capwap-saved-config-bak 82 -rwx 64 Dec 20 2016 09:47:15 +00:00 sensord_CSPRNG0 84 -rwx 64 Dec 20 2016 09:47:15 +00:00 sensord_CSPRNG1 10 drwx 2496 Dec 14 2016 13:29:02 +00:00 ap3g2-k9w8-mx.153-3.JC5 86 drwx 2496 Aug 31 2016 20:40:05 +00:00 ap3g2-k9w8-mx.153-3.JC3 40900608 bytes total (6049792 bytes free) Now here’s the file system of a 3702 that sows proper primary image, but will not allow a change to it’s backup software- which is stuck at 3.0.51.0. cst499c-6433#dir flash: Directory of flash:/ 2 -rwx 280 Jan 1 1970 00:03:07 +00:00 info XX 3 -rwx 965 Nov 18 2015 17:29:22 +00:00 lwapp_mm_mwar_hash.cfg 33 drwx 512 Mar 1 1993 00:03:32 +00:00 ap3g2-rcvk9w8-mx XX 4 -rwx 57080 Dec 20 2016 10:43:35 +00:00 event.log XX 6 -rwx 64 Dec 20 2016 10:43:28 +00:00 sensord_CSPRNG0 XX 72 drwx 2496 Dec 20 2016 10:43:10 +00:00 ap3g2-k9w8-mx.153-3.JC5 XX 66 drwx 0 Mar 1 1993 00:01:01 +00:00 configs XX 67 -rwx 64 Dec 20 2016 10:43:28 +00:00 sensord_CSPRNG1 XX 68 -rwx 128203 Nov 11 2014 17:15:38 +00:00 event.r1 8 -rwx 75 Mar 20 2017 13:30:24 +00:00 capwap-saved-config XX 9 -rwx 1950 May 26 2016 11:40:29 +00:00 atf_override_config_slot0.txt 7 -rwx 0 May 31 2016 08:35:40 +00:00 config.txt XX 11 -rwx 1950 May 26 2016 11:40:29 +00:00 atf_override_config_slot1.txt 10 -rwx 75 Mar 20 2017 13:33:45 +00:00 capwap-saved-config-bak XX 5 -rwx 12312 Mar 20 2017 13:33:45 +00:00 private-multiple-fs XX 12 -rwx 266 Dec 20 2016 10:45:59 +00:00 env_vars XX 40900608 bytes total (19593728 bytes free) This AP has lost it’s config in the past. Not sure what to do quite yet, and leery of yet another protracted TAC case. Lee Badman | Network Architect Adjunct Instructor | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e [email protected]<mailto:[email protected]> w its.syr.edu SYRACUSE UNIVERSITY syr.edu From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Mike Atkins Sent: Monday, March 20, 2017 9:31 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Lee, Are you thinking it works for a while but stops, as in needing to fsck the flash? Mike Atkins Network Engineer Office of Information Technology University of Notre Dame From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Lee H Badman Sent: Monday, March 20, 2017 9:08 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations This thread has me curious- I do see the occasional loss of AP name and config- like the AP goes back to default. Looking at our thousands of APs, I see some of the 7.1x backup code, but lots more of 3.0.51.0. I Tried to manually pull down newer backup image, and the process fails so I’m guessing there’s something amiss on the file system on these APs (have yet to look, anticipating not enough room in flash or similar). Lee Badman | Network Architect Adjunct Instructor | CWNE #200 Information Technology Services 206 Machinery Hall 120 Smith Drive Syracuse, New York 13244 t 315.443.3003 f 315.443.4325 e [email protected]<mailto:[email protected]> w its.syr.edu<http://its.syr.edu> SYRACUSE UNIVERSITY syr.edu<http://syr.edu> From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Mike Atkins Sent: Monday, March 20, 2017 6:45 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Same here, I’d say five in a three month period since upgrading from 8.2.121.0 to 8.2.141.0. Mike Atkins Network Engineer Office of Information Technology University of Notre Dame From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Jason Cook Sent: Monday, March 20, 2017 2:18 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Seems similar to what we have seen, reboots may or may not fix it and has been one of the few times where if at first you fail try the exact same thing over and over…… Basically we successfully resolved all issues with shut/unshut of ports sometimes up to 5x. Haven’t noticed the issue so much during operations but some software upgrades AP’s were like that. Heaps of AP’s showing the wrong backup image, thanks for the tip, will give it a try. First noticed on 3602is’. Hasn’t been a major problem but noticable -- Jason Cook Technology Services The University of Adelaide, AUSTRALIA 5005 Ph : +61 8 8313 4800 From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Daniel Brisson Sent: Friday, 17 March 2017 5:37 AM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Wanted to report that we also started seeing APs lose their hostname (and some lose their entire minds) around the time we went to 8.2. I just got off the phone with one of our techs who physically rebooted an AP and I’m now waiting to see if it will come back. When the AP is in the “bad state”, it shows up as a CDP neighbor on the switch as APxxxx.xxxx.xxxx, I can ping it, but ssh and telnet sessions are refused. I just looked and noticed a bunch of my APs show Backup SW version as 7.3.x, where most of them correctly show a Primary of 8.2.151.0 and a Backup of 8.2.131.40. I’m going to try the “Download Backup” to one of these APs to see if it fixes that. Thanks! -dan Dan Brisson Network Engineer University of Vermont From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:[email protected]] On Behalf Of Jeffrey D. Sessler Sent: Thursday, March 16, 2017 1:54 PM To: [email protected]<mailto:[email protected]> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations Ken, For the AP’s that have lost their name and require a reboot. Would you check the following for me? On WLC or PI, what do the problematic WAPs report as their backup software version? Typically, it should be the same as the “backup image” under commands->config boot on the controller. If it’s instead an older version e.g. 7.1.x, let me know. It’s circumstantial at this point, but I’ve noticed a pattern. * AP’s that exhibit the problem tend to also fail AP Image Pre-download (Download Primary) during code upgrades. If you make a note of these failures, those WAPs are more likely to have mental issues. * AP’s that exhibit the problem have very old (what shipped on it) code in the backup location e.g. 7.x * Issuing a AP Image Pre-download, Download Backup to these AP’s will replace the old code in the backup location. * Once the old backup image is updated, AP pre-download (Primary) now works during code upgrades, and the AP’s seem to stop losing their minds. Jeff From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> on behalf of Ken LeCompte <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, March 13, 2017 at 12:35 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [WIRELESS-LAN] Cisco WLC code recommendations We are currently running a handful of 5508s with 8.0.133.0 and have been stable for some time with around 400 APs and upwards of 1.5k clients. We also run a half dozen 5520s with 8.2.141.0 and they have been running solid with around 1k APs each and upwards of 10k clients. We do not however run anything but 2600, 3600, 2700 and 3700 APs. The only issue I have seen that I don’t understand well yet is related to some APs losing the minds during network interruptions. The APs will appear up from CDP neighbor information, but will have lost their name and will not connect to their configured primary or secondary controllers. A power cycle will often recover the AP, but not always. I believe that issue started with 8.2. Thank you. Ken -- Ken LeCompte - Consulting Telecommunications Analyst Telecommunications Division Office of Information Technology Rutgers, The State University of New Jersey Office ~ (848) 445-4823 On Mar 10, 2017, at 1:52 PM, Entwistle, Bruce <[email protected]<mailto:[email protected]>> wrote: We are currently running version 8.0.133.0 on our Cisco 5508 controllers, as our current access points are primarily 3500s and 3600s. However we have recently purchased a batch of 2802i access points whose minimum supported version is 8.2.110.0. I was looking to the group for their recommendations on a stable version of code which will support our new 2802i access points. Thank you Bruce Entwistle Network Manager University of Redlands ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/discuss.
