I actually looked into this but couldn’t find anything that made sense. Update to everyone: the problem is somehow solved. As I said we had 3 wlc’s, 2 hot, 1 standby. We moved AP’s from the failing wlc to the standby and everything started working like it was before the start of the academic year. I suspect some sort of a bug in the WLC where auth requests were put in a queue that wasn’t emptied or at a super slow pace.
Van: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Namens Carlo Terminiello Verzonden: woensdag 9 oktober 2019 9:28 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues Hi, Have you had a look at the AAA server statistics, will list number of auth requests, passes, fails, timeout etc.. example output below, may help focus the investigation. Of course a ‘debug client <mac>’ always helps Example output: (wlc01) >show radius auth statistics Authentication Servers: Server Index..................................... 1 Server Address................................... 10.203.251.110 Msg Round Trip Time.............................. 41087 (usec) Average Msg Round Trip Time...................... 154 (usec) Exponential Msg Round Trip Time.................. 37068 (usec) First Requests................................... 303910 Retry Requests................................... 42 Accept Responses................................. 22698 Reject Responses................................. 213 Challenge Responses.............................. 280986 Malformed Msgs................................... 0 Bad Authenticator Msgs........................... 0 Pending Requests................................. 0 Timeout Requests................................. 42 Consecutive Drops ............................... 0 Unknowntype Msgs................................. 0 Other Drops...................................... 13 AuthZ Requests................................... 0 AuthZ Accept Responses........................... 0 AuthZ Reject Responses........................... 0 --More-- or (q)uit Server Index..................................... 2 Server Address................................... 10.128.50.42 Msg Round Trip Time.............................. 154643 (usec) Average Msg Round Trip Time...................... 163837 (usec) Exponential Msg Round Trip Time.................. 208352 (usec) First Requests................................... 24776 Retry Requests................................... 34 Accept Responses................................. 24380 Reject Responses................................. 396 Challenge Responses.............................. 0 Malformed Msgs................................... 0 Bad Authenticator Msgs........................... 0 Pending Requests................................. 0 Timeout Requests................................. 34 Consecutive Drops ............................... 0 Unknowntype Msgs................................. 0 Other Drops...................................... 0 AuthZ Requests................................... 0 AuthZ Accept Responses........................... 0 AuthZ Reject Responses........................... 0 From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm <mathieu.st...@hogent.be<mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, 9 October 2019 at 08:11 To: <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471639471&sdata=JzaSRuy8XaNLidNS%2Fh9NENeDKgefU7m3XWMulJNpYoc%3D&reserved=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [https://www.hogent.be/www/assets/Image/logo2018.png] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 www.hogent.be<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471639471&sdata=chdIpiNbNoeCFJqFOwbUIJcMDlF06AHXwbDtYtPO1Qs%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471649465&sdata=XzYYwsvACoJjB2Kf15f0yw2HYNC9%2FL4lAYbgyCNmAKE%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471659460&sdata=OzxNVh7y952s2UhPkwfwzUf8Ca7NFg58kcG4uNbs8GQ%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471659460&sdata=OzxNVh7y952s2UhPkwfwzUf8Ca7NFg58kcG4uNbs8GQ%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity&data=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cdb9f08d74d59d13b%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062920471669453&sdata=aC4T0tLHgEgAX%2FVLyioI5%2FOPpuE964F81TaYB9bJ9c4%3D&reserved=0> ********** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
