RE: [WIRELESS-LAN] WLC & ISE combo issues
That was the reason I haven’t updated as well. I find it super confusing where everything went. Don’t know what they were thinking Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Heavrin, Lynn Verzonden: donderdag 10 oktober 2019 15:28 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues I’m sure you’re aware but you should skip 2.3 (super buggy) and go to 2.4, but the policy set UI has totally changed and in my opinion, is much, much harder to navigate than 2.2. That’s the only reason I’m holding off from upgrading 2.2 to 2.4. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Thursday, October 10, 2019 at 3:14 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues Thinking on going to latest ISE version (to get rid of that stupid flash ) when we have a new maintenance window. Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Heavrin, Lynn Verzonden: woensdag 9 oktober 2019 22:23 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues We have the same 5441 messages and we are on 8.5.135.0 and ISE 2.2 patch 12. I don’t have any evidence it’s service impacting but it is annoying. You need to upgrade from patch 5 to address some serious bug and vulnerabilities. Patch 15 is out. We also get the 5441 messages on our VPN auth on ISE so it’s not isolated to wifi. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Kitri Waterman mailto:wate...@wwu.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, October 9, 2019 at 10:17 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues 8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fwireless%2Fwireless-lan-controller-software%2F200046-tac-recommended-aireos.html%23anc13=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C349ddc42f3984ebea74708d74d85bee4%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637063109151199539=f6VSxWBxRTpDzraZLbpVPUz9NP0kD7GT%2FsGCzySJpPE%3D=0> Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.pr
Re: [WIRELESS-LAN] WLC & ISE combo issues
I’m sure you’re aware but you should skip 2.3 (super buggy) and go to 2.4, but the policy set UI has totally changed and in my opinion, is much, much harder to navigate than 2.2. That’s the only reason I’m holding off from upgrading 2.2 to 2.4. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Mathieu Sturm Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Thursday, October 10, 2019 at 3:14 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues Thinking on going to latest ISE version (to get rid of that stupid flash ) when we have a new maintenance window. Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Heavrin, Lynn Verzonden: woensdag 9 oktober 2019 22:23 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues We have the same 5441 messages and we are on 8.5.135.0 and ISE 2.2 patch 12. I don’t have any evidence it’s service impacting but it is annoying. You need to upgrade from patch 5 to address some serious bug and vulnerabilities. Patch 15 is out. We also get the 5441 messages on our VPN auth on ISE so it’s not isolated to wifi. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Kitri Waterman mailto:wate...@wwu.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, October 9, 2019 at 10:17 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues 8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fwireless%2Fwireless-lan-controller-software%2F200046-tac-recommended-aireos.html%23anc13=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C0e0d9fa7f9b84cb5569908d74cf68a44%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062494090192545=1Wea7FcwIHYXTDfd66dK2jonTcxZBlPyzurrvBdd84k%3D=0> Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C0e0d9fa7f9b84cb5569908d74cf68a44%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062494090192545=OxOm2kKVpG%2FKEQw7McWOqZZP2cGg9o9yaa8ZphNwDw4%3D=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto
RE: [WIRELESS-LAN] WLC & ISE combo issues
I actually looked into this but couldn’t find anything that made sense. Update to everyone: the problem is somehow solved. As I said we had 3 wlc’s, 2 hot, 1 standby. We moved AP’s from the failing wlc to the standby and everything started working like it was before the start of the academic year. I suspect some sort of a bug in the WLC where auth requests were put in a queue that wasn’t emptied or at a super slow pace. Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Carlo Terminiello Verzonden: woensdag 9 oktober 2019 9:28 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues Hi, Have you had a look at the AAA server statistics, will list number of auth requests, passes, fails, timeout etc.. example output below, may help focus the investigation. Of course a ‘debug client ’ always helps Example output: (wlc01) >show radius auth statistics Authentication Servers: Server Index. 1 Server Address... 10.203.251.110 Msg Round Trip Time.. 41087 (usec) Average Msg Round Trip Time.. 154 (usec) Exponential Msg Round Trip Time.. 37068 (usec) First Requests... 303910 Retry Requests... 42 Accept Responses. 22698 Reject Responses. 213 Challenge Responses.. 280986 Malformed Msgs... 0 Bad Authenticator Msgs... 0 Pending Requests. 0 Timeout Requests. 42 Consecutive Drops ... 0 Unknowntype Msgs. 0 Other Drops.. 13 AuthZ Requests... 0 AuthZ Accept Responses... 0 AuthZ Reject Responses... 0 --More-- or (q)uit Server Index. 2 Server Address... 10.128.50.42 Msg Round Trip Time.. 154643 (usec) Average Msg Round Trip Time.. 163837 (usec) Exponential Msg Round Trip Time.. 208352 (usec) First Requests... 24776 Retry Requests... 34 Accept Responses. 24380 Reject Responses. 396 Challenge Responses.. 0 Malformed Msgs... 0 Bad Authenticator Msgs... 0 Pending Requests. 0 Timeout Requests. 34 Consecutive Drops ... 0 Unknowntype Msgs. 0 Other Drops.. 0 AuthZ Requests... 0 AuthZ Accept Responses... 0 AuthZ Reject Responses... 0 From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, 9 October 2019 at 08:11 To: mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7Ccb67af74b14b420cd
RE: [WIRELESS-LAN] WLC & ISE combo issues
Yes Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Jonathan Oakden Verzonden: donderdag 10 oktober 2019 10:25 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues Are you using PEAP/MSCHAP? From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply to: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Tuesday, 8 October 2019 at 20:00 To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [https://www.hogent.be/www/assets/Image/logo2018.png] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 www.hogent.be<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C81978c8f734e4806438f08d74d5b5fd5%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062927155163456=0YQYXA10zbW7NZxE3g5SJR34deS%2F%2FGU7ceN6yU%2BKJFg%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C81978c8f734e4806438f08d74d5b5fd5%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062927155173453=FhehFOmaBbnLbw58aBAb8JzJ8kgKDM%2Fo3rFPOEggIxM%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C81978c8f734e4806438f08d74d5b5fd5%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062927155173453=FhehFOmaBbnLbw58aBAb8JzJ8kgKDM%2Fo3rFPOEggIxM%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] WLC & ISE combo issues
Are you using PEAP/MSCHAP? From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Mathieu Sturm Reply to: The EDUCAUSE Wireless Issues Community Group Listserv Date: Tuesday, 8 October 2019 at 20:00 To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [https://www.hogent.be/www/assets/Image/logo2018.png] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 www.hogent.be<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F=02%7C01%7Cmathieu.sturm%40hogent.be%7C86879fbc6e8c49ab13ff08d67ac4edef%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C636831383554731873=8NfYjNEE4XDViDT6wMtCYFa0cY8g5CXqS9kf7VtYBcU%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] WLC & ISE combo issues
Thinking on going to latest ISE version (to get rid of that stupid flash ) when we have a new maintenance window. Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Heavrin, Lynn Verzonden: woensdag 9 oktober 2019 22:23 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues We have the same 5441 messages and we are on 8.5.135.0 and ISE 2.2 patch 12. I don’t have any evidence it’s service impacting but it is annoying. You need to upgrade from patch 5 to address some serious bug and vulnerabilities. Patch 15 is out. We also get the 5441 messages on our VPN auth on ISE so it’s not isolated to wifi. From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Kitri Waterman mailto:wate...@wwu.edu>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Wednesday, October 9, 2019 at 10:17 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues 8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fwireless%2Fwireless-lan-controller-software%2F200046-tac-recommended-aireos.html%23anc13=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C0e0d9fa7f9b84cb5569908d74cf68a44%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062494090192545=1Wea7FcwIHYXTDfd66dK2jonTcxZBlPyzurrvBdd84k%3D=0> Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C0e0d9fa7f9b84cb5569908d74cf68a44%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062494090192545=OxOm2kKVpG%2FKEQw7McWOqZZP2cGg9o9yaa8ZphNwDw4%3D=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K ass
Re: [WIRELESS-LAN] WLC & ISE combo issues
Hi, Have you had a look at the AAA server statistics, will list number of auth requests, passes, fails, timeout etc.. example output below, may help focus the investigation. Of course a ‘debug client ’ always helps Example output: (wlc01) >show radius auth statistics Authentication Servers: Server Index. 1 Server Address... 10.203.251.110 Msg Round Trip Time.. 41087 (usec) Average Msg Round Trip Time.. 154 (usec) Exponential Msg Round Trip Time.. 37068 (usec) First Requests... 303910 Retry Requests... 42 Accept Responses. 22698 Reject Responses. 213 Challenge Responses.. 280986 Malformed Msgs... 0 Bad Authenticator Msgs... 0 Pending Requests. 0 Timeout Requests. 42 Consecutive Drops ... 0 Unknowntype Msgs. 0 Other Drops.. 13 AuthZ Requests... 0 AuthZ Accept Responses... 0 AuthZ Reject Responses... 0 --More-- or (q)uit Server Index. 2 Server Address... 10.128.50.42 Msg Round Trip Time.. 154643 (usec) Average Msg Round Trip Time.. 163837 (usec) Exponential Msg Round Trip Time.. 208352 (usec) First Requests... 24776 Retry Requests... 34 Accept Responses. 24380 Reject Responses. 396 Challenge Responses.. 0 Malformed Msgs... 0 Bad Authenticator Msgs... 0 Pending Requests. 0 Timeout Requests. 34 Consecutive Drops ... 0 Unknowntype Msgs. 0 Other Drops.. 0 AuthZ Requests... 0 AuthZ Accept Responses... 0 AuthZ Reject Responses... 0 From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Mathieu Sturm Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Wednesday, 9 October 2019 at 08:11 To: Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worry
RE: [WIRELESS-LAN] WLC & ISE combo issues
There is a 8.5 MR5 since June. Any known major issues on that? Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Kitri Waterman Verzonden: woensdag 9 oktober 2019 17:17 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues 8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fwireless%2Fwireless-lan-controller-software%2F200046-tac-recommended-aireos.html%23anc13=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C6a49a279d91d45665d1108d74ccbb979%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062310199002274=vTqb189Y8NHSVK7DJ%2B0zDT08wQiFiS%2FFLgk4CAO4P1M%3D=0> Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> on behalf of Mathieu Sturm mailto:mathieu.st...@hogent.be>> Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C6a49a279d91d45665d1108d74ccbb979%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062310199012272=TB6z94eD94rF%2FPsB5fz%2BRH%2FZ5BGhd0ugsXJdtlyPCqs%3D=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [https://www.hogent.be/www/assets/Image/logo2018.png] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B
RE: [WIRELESS-LAN] WLC & ISE combo issues
This is a setup that's around for some time. Definitely not something new. We might have around 500 users more than last year but I think this couldn't make a lot of difference. -Oorspronkelijk bericht- Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Dennis Xu Verzonden: woensdag 9 oktober 2019 15:20 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues Is this a new deployment or do you have more users this year than last year? It could be load related. That 5441 error log indicates there are queued RADIUS packets at ISE which cannot be processed in timely manner. Try adding ISE service node to see if that can help. Also check this link about something to be tuned at WLC side: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fc%2Fen%2Fus%2Fsupport%2Fdocs%2Fwireless-mobility%2Fwireless-lan-wlan%2F118703-technote-wlc-00.htmldata=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C08b3e497408d4872b44a08d74cbb5da9%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062239935089762sdata=agnSRfe0pr2Z%2FgsHPU1LCc%2F3GFIMO4ovD9kDpVme5s8%3Dreserved=0. Cheers, Dennis Xu | Analyst III, Network Infrastructure Computing and Communications Services (CCS) | University of Guelph University Centre | 50 Stone Rd E | Guelph, ON | N1G 2W1 519-824-4120 Ext. 56217 | d...@uoguelph.ca https://eur03.safelinks.protection.outlook.com/?url=www.uoguelph.ca%2Fccsdata=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C08b3e497408d4872b44a08d74cbb5da9%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062239935089762sdata=db9UP6fXQhd%2FoCiqlrW3%2FUlZaZMJJ02wt3xLjw%2FJFGU%3Dreserved=0 | twitter.com/ccsnews | facebook.com/CCSUofG -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Kenny, Eric Sent: Wednesday, October 9, 2019 9:09 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues Hi Mathieu, One thing you might want to verify is that the RADIUS timeout values match in both the WLCs and in ISE. If these values differ, you may end up in a situation like this where one side gives up and the other side is not aware. --- Eric Kenny Network Architect Harvard University ITS --- > On Oct 8, 2019, at 2:50 PM, Mathieu Sturm wrote: > > Hello, since the start of the new academic year we’ve been having some > troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is > standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and > 3 radius-only nodes). > > We have this setup since 2018. There were some problems sometimes but nothing > major. Now recently it’s taking a long time for people to get connected. We > have around 20k students and 3K staff with peaks to nearly 9K associations. > > The problem is that it is difficult to get connected sometimes. I see the > user trying to connect in the WLC’s but don’t see them trying in the ISE’s > (it looks like the attempt gets lost somewher). > I can see the following worrying log message in the wlc: > > RADIUS auth-server X.X.X.X unavailable > > Or > > These logs in the ISE > > 5441 Endpoint started new session while the packet of previous session is > being processed. Dropping new session. > 12930 Supplicant stopped responding to ISE after sending it the first > PEAP message > > > It looks like there is some sort of bottleneck between WLC and ISE. > > Further information: the identity store is a bunch of Windows Domain > Controllers (6 in total). > > Any ideas? > > Mathieu Sturm > Hoofdmedewerker Netwerkbeheer > > > > Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus > Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 > Gent > +32 9 243 35 23 > https://eur03.safelinks.protection.outlook.com/?url=www.hogent.bedata=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C08b3e497408d4872b44a08d74cbb5da9%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062239935089762sdata=6%2BB0NUZS7MVoXPa29lKkNx1s0rVc4xXUpmeLAyKxg%2FY%3Dreserved=0 > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email > reply. Additional participation and subscription information can be > found at > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=02%7C01%7Cmathieu.sturm%40HOGENT.BE%7C08b3e497408d4872b44a08d74cbb5da9%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C637062239935089762sdata=M7P%2FQmDPCm%2Bp07wwV2rqGqCGk%2F9oNxjyu2c10Ihjj34%3Dreserved=0 > ** Replies to EDUCAUSE Community Group emails are sen
Re: [WIRELESS-LAN] WLC & ISE combo issues
We have the same 5441 messages and we are on 8.5.135.0 and ISE 2.2 patch 12. I don’t have any evidence it’s service impacting but it is annoying. You need to upgrade from patch 5 to address some serious bug and vulnerabilities. Patch 15 is out. We also get the 5441 messages on our VPN auth on ISE so it’s not isolated to wifi. From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Kitri Waterman Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Wednesday, October 9, 2019 at 10:17 AM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues 8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13 Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Mathieu Sturm Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cwatermk%40WWU.EDU%7C90f64cd5ebf74de1e21e08d74c7f87a0%7Cdc46140ce26f43efb0ae00f257f478ff%7C0%7C0%7C637061982933539827=d6N7J8%2BnK19EbRL2tx4W3%2Bnzkwbk8sBPEjBvfW4Leh8%3D=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [cid:image001.png@01D57EB5.7BF03DA0] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 www.hogent.be<https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F=02%7C01%7Cwatermk%40WWU.EDU%7C90f64cd5ebf74de1e21e08d74c7f87a0%7Cdc46140ce26f43efb0ae00f257f478ff%7C0%7C0%7C637061982933549823=6T5ZRTNKQdlY93PbxuFKifDX%2FYdJIEWjQNx6%2FwTmxG0%3D=0> ** Replies to EDUCAUSE Community Group emails are s
Re: [WIRELESS-LAN] WLC & ISE combo issues
8.3.x? Or 8.5.x? 8.5 will support AP2600’s. We’re currently at 8.5.140.0 (we still have AP3500’s to support…) and it’s been fairly stable for AireOS. 8.3 also has some escalation fixes: https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc13 Kitri Network Architect/Engineer Enterprise Infrastructure Services Western Washington University From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Mathieu Sturm Reply-To: The EDUCAUSE Wireless Issues Community Group Listserv Date: Tuesday, October 8, 2019 at 11:11 PM To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues The WLC is on version 8.3.140.0 (we still have 2600 series AP’s that we need to replace so we are pretty limited) and ISE is 2.2 (patch 5). Van: The EDUCAUSE Wireless Issues Community Group Listserv Namens Letts, Richard J Verzonden: dinsdag 8 oktober 2019 22:41 Aan: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Onderwerp: Re: [WIRELESS-LAN] WLC & ISE combo issues What version of core on the WLC / what model of AP? We had an issue at the start of the year with version of code on cisco 3500 series AP where clients would successful authenticate with the AP, but the association would never get passed from the AP through to the controller and thence on to the ISE. Clients would get a ‘bad password’ (or similar type of error) displayed on their computer which would confuse them, and there would be nothing recorded in the WLC or ISE logs. Authentication and Association isn’t the way around people normally think of this. https://documentation.meraki.com/MR/WiFi_Basics_and_Best_Practices/802.11_Association_process_explained<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocumentation.meraki.com%2FMR%2FWiFi_Basics_and_Best_Practices%2F802.11_Association_process_explained=02%7C01%7Cwatermk%40WWU.EDU%7C90f64cd5ebf74de1e21e08d74c7f87a0%7Cdc46140ce26f43efb0ae00f257f478ff%7C0%7C0%7C637061982933539827=d6N7J8%2BnK19EbRL2tx4W3%2Bnzkwbk8sBPEjBvfW4Leh8%3D=0> anyway, I think you’re going to need to include version numbers of the ISE and WLC code for more help. Thank you Richard Letts From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Mathieu Sturm Sent: Tuesday, October 8, 2019 2:50 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer [https://www.hogent.be/www/assets/Image/logo2018.png] Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 www.hogent.be<https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F=02%7C01%7Cwatermk%40WWU.EDU%7C90f64cd5ebf74de1e21e08d74c7f87a0%7Cdc46140ce26f43efb0ae00f257f478ff%7C0%7C0%7C637061982933549823=6T5ZRTNKQdlY93PbxuFKifDX%2FYdJIEWjQNx6%2FwTmxG0%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunity=02%7C01%7Cwatermk%40WWU.EDU%7C90f64cd5ebf74de1e21e08d74c7f87a0%7Cdc46140ce26f43efb0ae00f257f478ff%7C0%7C0%7C637061982933559812=auTU76erLyyypH0JeKLCEL0aHbiWcx6wLkZVpInu1YQ%3D=0> ** Replies to EDUCAUSE Community Group emails are sent to the entire communit
RE: [WIRELESS-LAN] WLC & ISE combo issues
Is this a new deployment or do you have more users this year than last year? It could be load related. That 5441 error log indicates there are queued RADIUS packets at ISE which cannot be processed in timely manner. Try adding ISE service node to see if that can help. Also check this link about something to be tuned at WLC side: https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/118703-technote-wlc-00.html. Cheers, Dennis Xu | Analyst III, Network Infrastructure Computing and Communications Services (CCS) | University of Guelph University Centre | 50 Stone Rd E | Guelph, ON | N1G 2W1 519-824-4120 Ext. 56217 | d...@uoguelph.ca www.uoguelph.ca/ccs | twitter.com/ccsnews | facebook.com/CCSUofG -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Kenny, Eric Sent: Wednesday, October 9, 2019 9:09 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] WLC & ISE combo issues Hi Mathieu, One thing you might want to verify is that the RADIUS timeout values match in both the WLCs and in ISE. If these values differ, you may end up in a situation like this where one side gives up and the other side is not aware. --- Eric Kenny Network Architect Harvard University ITS --- > On Oct 8, 2019, at 2:50 PM, Mathieu Sturm wrote: > > Hello, since the start of the new academic year we’ve been having some > troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is > standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and > 3 radius-only nodes). > > We have this setup since 2018. There were some problems sometimes but nothing > major. Now recently it’s taking a long time for people to get connected. We > have around 20k students and 3K staff with peaks to nearly 9K associations. > > The problem is that it is difficult to get connected sometimes. I see the > user trying to connect in the WLC’s but don’t see them trying in the ISE’s > (it looks like the attempt gets lost somewher). > I can see the following worrying log message in the wlc: > > RADIUS auth-server X.X.X.X unavailable > > Or > > These logs in the ISE > > 5441 Endpoint started new session while the packet of previous session is > being processed. Dropping new session. > 12930 Supplicant stopped responding to ISE after sending it the first > PEAP message > > > It looks like there is some sort of bottleneck between WLC and ISE. > > Further information: the identity store is a bunch of Windows Domain > Controllers (6 in total). > > Any ideas? > > Mathieu Sturm > Hoofdmedewerker Netwerkbeheer > > > > Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus > Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 > Gent > +32 9 243 35 23 > www.hogent.be > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email > reply. Additional participation and subscription information can be > found at https://www.educause.edu/community > ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] WLC & ISE combo issues
Hi Mathieu, One thing you might want to verify is that the RADIUS timeout values match in both the WLCs and in ISE. If these values differ, you may end up in a situation like this where one side gives up and the other side is not aware. --- Eric Kenny Network Architect Harvard University ITS --- > On Oct 8, 2019, at 2:50 PM, Mathieu Sturm wrote: > > Hello, since the start of the new academic year we’ve been having some > troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is > standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and > 3 radius-only nodes). > > We have this setup since 2018. There were some problems sometimes but nothing > major. Now recently it’s taking a long time for people to get connected. We > have around 20k students and 3K staff with peaks to nearly 9K associations. > > The problem is that it is difficult to get connected sometimes. I see the > user trying to connect in the WLC’s but don’t see them trying in the ISE’s > (it looks like the attempt gets lost somewher). > I can see the following worrying log message in the wlc: > > RADIUS auth-server X.X.X.X unavailable > > Or > > These logs in the ISE > > 5441 Endpoint started new session while the packet of previous session is > being processed. Dropping new session. > 12930 Supplicant stopped responding to ISE after sending it the first PEAP > message > > > It looks like there is some sort of bottleneck between WLC and ISE. > > Further information: the identity store is a bunch of Windows Domain > Controllers (6 in total). > > Any ideas? > > Mathieu Sturm > Hoofdmedewerker Netwerkbeheer > > > > Directie Financiën, Infrastructuur en IT > Afdeling Netwerkbeheer > Campus Schoonmeerssen - Gebouw B Lokaal B0.75 > Valentin Vaerwyckweg 1 - 9000 Gent > +32 9 243 35 23 > www.hogent.be > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire community > list. If you want to reply only to the person who sent the message, copy and > paste their email address and forward the email reply. Additional > participation and subscription information can be found at > https://www.educause.edu/community > ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [WIRELESS-LAN] WLC & ISE combo issues
Mathieu, What version of ISE and WLC are you running? We had a memory leak in ISE 2.6 which was causing latency. About a month ago, we patched and then had TAC do a manual cleanup of the db. So far so good. Christina Klam Network Engineer Institute for Advanced Study 1 Einstein Dr Princeton, NJ 08540 +1 609-734-8154 ck...@ias.edu From: "Mathieu Sturm" To: "The EDUCAUSE Wireless Issues Community Group Listserv" Sent: Tuesday, October 8, 2019 2:50:13 PM Subject: [WIRELESS-LAN] WLC & ISE combo issues Hello, since the start of the new academic year we’ve been having some troubles with our Cisco setup. We have 3 Cisco WLC 5520’s (one of these is standby), around 850ap’s and 5 Cisco ISE’s (1 admin node, 1 monitor node and 3 radius-only nodes). We have this setup since 2018. There were some problems sometimes but nothing major. Now recently it’s taking a long time for people to get connected. We have around 20k students and 3K staff with peaks to nearly 9K associations. The problem is that it is difficult to get connected sometimes. I see the user trying to connect in the WLC’s but don’t see them trying in the ISE’s (it looks like the attempt gets lost somewher). I can see the following worrying log message in the wlc: RADIUS auth-server X.X.X.X unavailable Or These logs in the ISE 5441 Endpoint started new session while the packet of previous session is being processed. Dropping new session. 12930 Supplicant stopped responding to ISE after sending it the first PEAP message It looks like there is some sort of bottleneck between WLC and ISE. Further information: the identity store is a bunch of Windows Domain Controllers (6 in total). Any ideas? Mathieu Sturm Hoofdmedewerker Netwerkbeheer Directie Financiën, Infrastructuur en IT Afdeling Netwerkbeheer Campus Schoonmeerssen - Gebouw B Lokaal B0.75 Valentin Vaerwyckweg 1 - 9000 Gent +32 9 243 35 23 [ https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.hogent.be%2F=02%7C01%7Cmathieu.sturm%40hogent.be%7C86879fbc6e8c49ab13ff08d67ac4edef%7C5cf7310e091a4bc5acd726c721d4cccd%7C1%7C0%7C636831383554731873=8NfYjNEE4XDViDT6wMtCYFa0cY8g5CXqS9kf7VtYBcU%3D=0 | www.hogent.be ] ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at [ https://www.educause.edu/community | https://www.educause.edu/community ] ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
