-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
We are running Cisco WISM-2 controllers, code 7.4.110.0 at 11AM this morning all our clients were dropped and could not reconnect. To get the clients back online we manually failed all access points to a secondary controller. This worked great for about 30 minutes then all the clients were dropped off again. At this point we experienced a total wireless outage. In the logs we see the following errors: Sep 25 12:52:55.726: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 08:cc:68:62:9d:40 Sep 25 12:52:55.727: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 08:cc:68:62:9d:40 Sep 25 12:52:55.727: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 70:10:5c:b9:7e:a0 Sep 25 12:52:55.728: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 08:cc:68:b8:58:b0 Sep 25 12:52:55.728: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 08:cc:68:62:98:e0 Sep 25 12:52:55.728: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 08:cc:68:b8:58:b0 Sep 25 12:52:55.728: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 08:cc:68:62:98:e0 Sep 25 12:52:55.729: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP f4:1f:c2:02:48:f0 Sep 25 12:52:55.730: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP f4:1f:c2:02:48:f0 Sep 25 12:52:55.730: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 70:10:5c:b9:73:c0 Sep 25 12:52:55.730: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP f4:1f:c2:02:48:f0 Sep 25 12:52:55.731: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 08:cc:68:62:98:e0 Sep 25 12:52:55.731: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 08:cc:68:62:e7:90 Sep 25 12:52:55.733: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP f4:1f:c2:03:2b:f0 Sep 25 12:52:55.733: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 08:cc:68:62:8a:10 Sep 25 12:52:55.734: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP f4:1f:c2:98:99:20 Sep 25 12:52:55.735: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 08:cc:68:62:8a:10 Sep 25 12:52:55.736: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 08:cc:68:62:93:c0 Sep 25 12:52:55.737: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 08:cc:68:b8:66:50 Sep 25 12:52:55.737: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 70:10:5c:b9:7e:a0 Sep 25 12:52:55.737: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 70:10:5c:b9:7e:a0 Sep 25 12:52:55.738: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 20:bb:c0:e7:8f:60 Sep 25 12:52:55.738: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11712 The system is unable to send delete mobile message to LWAPP; AP 44:ad:d9:61:f8:c0 Sep 25 12:52:55.738: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 08:cc:68:b8:68:a0 Sep 25 12:52:55.738: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 20:bb:c0:e7:8f:60 Sep 25 12:52:55.738: #LWAPP-3-MSG_SEND_ERR: spam_api.c:1489 The system is unable to send free AID message to LWAPP; AP 44:ad:d9:61:f8:c0 Sep 25 12:52:55.739: #LWAPP-3-MSG_SEND_ERR: spam_api.c:11616 The system is unable to send add mobile message to LWAPP; AP 20:bb:c0:e7:8f:60 In order to get something back online we rebooted one of the WISMs, failed 50% of the AP's back to the newly rebooted WISM. At this point both controllers started working. TAC was online with us the entire time. They are still reviewing the logs, but at this point they have no idea what the problem is (was). Controller uptime was 22 days, memory around 30-40%, cpu 5-10%, 5000-6000 active clients. This just happened, we are still reviewing all the logs, graphs, stats, etc... I wanted to post this up right now just to see if anyone else has experienced an issue like this. Once TAC gets back with us, I will post additional updates. Thanks, - -- Jason E. Murray Sr. Systems Engineer Washington University in St. Louis Phone: 314-935-4865 Email: [email protected] Web: http://nts.wustl.edu/~jemurray/ -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJSQzHhAAoJEG6KT83y+ROMcAQQAKOIp/ReGDzwGBPcSmbubADn wDHj5TA84Ay1Vk3Be4SHHfGScjQcrWMuFyO/yzPt35RyIZ9Vno4VgkCpncvBeJGb SD5LVaGb+JQn/bRplxvoE4ttjL7Sh1WcbkgmD9yS6jhAi8FmcS2Wu55m0d1rDWWO Z32ivf0YSkywbVqI/ySws6Rq+kYtEWmh8qYard8J/0b6BVm7S+k4XweBP1b/wb0q RbHBWj7KJ9rW9zMAS803UudVBwU8j1Cn37KSDM4VLWd3Z7pz7xcZOWoQ3ilc6Bjc 4OzQtsuPBhyaJkVDbAnjvhDW98t3y4Emp8lUPqdXOuhM+GI1rbbhOx8xoIvbF9Gw C7x/VbuF8CIlVhoBHriAMw4z+PGTKBEHt4MMo7w790aub8LVncfAwnzY47tsMIjP 4B0VGGZpP98kQ/imUpbYk7MMb822Wbgpm7FRimQPLUkmSpu9g5ez9S0YdPjWO2Lh 23AG0dv13Ht+Z+zJxzf9YXSOIQlypW2N6KXZD/lFemUlDVj2guJXZgCeRtS4rk0y 1i27FhgvZnUWjmKyqNMBfr4AhotnZts+7j8gVmvUuFdn6S2xrpFa2npLznkJywU+ r20NBOUPC0SEJNn0u4HIanwOrK1XnM2CYgKpCRKwewtmXhW6GZ5DzHupfI/Xq1dC 7bAgWJ2eYFxAfBpHYx66 =D9Ih -----END PGP SIGNATURE----- ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
