PoE Load Tester Recommendation

[Floyd, Brad]

Can anyone recommend a device to PoE load test network jacks? I have some jacks 
that pass the installer's Category Certification, but are not passing the 
appropriate PoE to bring the APs online. I would like to be able to load test 
for 802.3af, 802.3at, and 802.3bt (at both 60W and 90W), as appropriate. I 
assume I would need to be able to set the load to apply (in Watts) and see the 
voltage level at the Powered Device. The usual constraints apply. Cheaper, but 
reliable is best.
Thanks,
Brad


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: rough start of semester on 9800-80 WLCs

[Rios, Hector J]

There are two commands you can use:

show wireless loadbalance tag affinity wncd 
show wireless stats ap loadbalance summary

Hector Rios
UT Austin


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 1:48 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Thanks for the info.  I'll look into the AP service pack.  We haven't done one 
of those yet so kind of curious to see it in action.

Yeah we had mixed results with the 500 APs in a site tag.  Some of the areas on 
campus were fine.  I think client count had something to do with provoking it.  
Our highest population areas were the ones that saw the most capwap timeouts.  
Just curious- how are you checking the number of APs and site tags assigned to 
a wncd process?

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Rios, Hector J
Sent: Tuesday, September 7, 2021 12:57 PM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Chad,

Sorry to hear about the issues you ran into. We also started the semester with 
9800-80s, but we chose to go with 17.3.4.

Things went well for most of the day on the first day of classes, except for a 
single controller crash after business hours. Cisco has identified this as a 
bug on the 17.3.X:
CSCvx71141 - CPU HOG in RRM Process.

You should contact TAC to get more details. They might also be able to provide 
a workaround, depending on your configuration.

We also ran into the bug below, but this was fixed with an AP service pack. 
Cool feature BTW, it actually works.
CSCvz08781
Symptom: AP2800/3800/4800/1560/IW6300/ESW6300 Firmware Radio Crash on 17.3.4 
while passing client traffic.

There is also an issue on 17.3.4 that is impacting 9120s. Cisco is working on a 
service pack for this as well. Don't have more details on this.

Thank you on the information regarding the wncd processes. We also followed the 
best practices, but we do have controllers that have a few wncd processes with 
a little over 500 APs. No issues so far, other than we have noticed in a few 
instances that even though we only have 8 custom site tags, some WLCs will 
assign two sitetags to a single wncd process.  We are working with TAC on this.

We also have a substantial number of 2700 series AP. We encountered no major 
issues during the upgrade process.

Finally, we have noticed that L3 roaming is not working on our 802.1X and PSK 
SSIDs. I wonder if anyone has run into this issue as well?.

Best,

Hector Rios
UT Austin




From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 9:21 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.4.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.4.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 

Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Travis Schick]

Have seen similar behavior and strongly recommend using validuser acl  at
very least change it form default any any- can start small and deny/protect
critical IP's in your infrastructure

its all fun and games until a user device gets picked up as your DNS server
or local ip gateway

but would recommend ultimately making validuser acl only accept ip's you
expect your client to have

when it's happening it sure seems malicious - but have learned not to
assign intent to most actions of my users.

On Tue, Sep 7, 2021 at 12:53 PM Johnson, Christopher 
wrote:

> Sid,
>
>
> We know from personal experience of running into this issue several years
> ago. Like David, we’ve instituted a few validuserACLs – (I actually use
> aliases for those subnets – so that I can re-use them in other places and
> to give a description of those valid ip addresses).
>
> After finding the offending device, was 99% positive it was malicious –
> but as I dived into the Rabbit Hole – discovered it was just a stupid
> malfunctioning device…a Roku Stick. I’ve also seen this behavior on other
> devices that make use of a “Router/IP Sharing” SSID such as “Roku’s Dorm
> Mode” or “Internet Sharing” with Windows.
>
> The Roku generates it’s own SSID “AP Mode” while connecting to our
> infrastructure SSID – it’s not bridged – but routed based on the fact that
> when you connect your phone or computer to the Roku’s SSID – your assigned
> a 192.168.X private IP Address. What I suspect happened in our scenario
> (I’ll use your 23.185.0.1 address for example).
>
> 1. Student Connected Roku to Guest SSID
>
> 2. Roku Prompted Student to use “Dorm Mode”
>
> 3. Student Connected to Roku with iPhone or Computer with a “home page” of
> our institution’s website.
>
> 4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 –
> Destination: 192.168.X.X – instead of sending it to the “private network”
> wifi interface  to the user’s iPhone or computer – it sent it out the
> “infrastructure network” interface – which based on how a “User” gets into
> the table à
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
> – and was assigned the guest unauthenticated policy – denying all traffic –
> cept icmps.
>
>
>
> I first started suspecting things weren’t as “simple” as they may be when
> I noticed Roku’s were “claiming” the IP Addresses of Google – what was
> funny was seeing the Controller prevent one Roku from entering the
> User-Table with a Google IP Address – *ONLY because another Roku* had
> already sourced a packet with Google’s IP Address.
>
>
>
> If you add a “any any any deny” with “LOG” option enabled – you can see
> ALL the invalid sessions that would have entered the user-table – including
> their destinations.
>
>
> I was only able to “partially replicate the behavior” – but it’s still a
> strong case.
>
> A few links down below:
>
>
> How the user gets into the user-table of the controller? -
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
> IP Address Leaking -
> https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e
>
>
> Some info from the ArubaOS Hardening Guide
>
> https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364
> - Page 26 and 27 goes into detail about “validuser” and
> “local-valid-users” – “local-valid-users” requires the controller to have
> an IP Address on that VLAN interface. There’s also the “Enforce DHCP”
> option in each AAA Aruba Profile – essentially a per SSID setting.
>
>
>
>
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c
>
>
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4
>
>
>
> http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/
> -à (BROKEN LINK Now ☹)
> Unfortunately the video link I had from commsolutions – they had
> presentation demonstrating this issue but it’s a broken link now –one of
> their customers for whatever reason had their guests manually enter the ip
> addresses onto their ipads – and someone flip-flopped the “IP Address” and
> the “Default Gateway”….started denying traffic for the default
> gateway….whoops!
>
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Mike Fitzgerald
> *Sent:* Tuesday, September 07, 2021 12:16 PM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> *Subject:* Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites
> inaccessible from wireless network - Aruba
>
>
>
> Some people who received this message don't often get email from
> fi...@brandeis.edu. Learn why this is important
> 

Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Daniel Westacott]

Hi Educause wifi:

We use a filter that only allows clients to "have" a valid IP address from
"our" range.
It' a bit of overhead, but it solves this issue for us.  We also say
clients listed with addresses that really make no sense.

you build a list something like this:

netdestination umn-wiredv4-wireless-user-networks
network 10.128.0.0 255.224.0.0
network 10.160.0.0 255.240.0.0
network 192.168.157.0 255.255.255.192
network 10.32.253.128 255.255.255.128
network 10.33.9.0 255.255.255.0
description "wiredv4 service ip's for users"

add it to valid user:

ip access-list session validuser
network 127.0.0.0 255.0.0.0 any any deny
network 169.254.0.0 255.255.0.0 any any deny
network 224.0.0.0 240.0.0.0 any any deny
host 255.255.255.255 any any deny
network 240.0.0.0 240.0.0.0 any any deny
alias umn-wiredv4-wireless-user-networks any any permit
any any any deny

Something similar is needed for V6.
/daniel/
daniel westacott
University of Minnesota



On Tue, Sep 7, 2021 at 11:04 AM Sidharth Nandury 
wrote:

> So. sigh!
>
> It seems like an end client either statically or for some unknown reason
> got assigned the IP address for these websites. The role that the client
> was assigned had a policy to "deny" traffic to the internet (as per
> design). The part that we did not know was that when a client is going to a
> particular destination, the controllers look at the user table to see if
> there is an IP and a route available before even going to the role-based
> ACLs.
>
> Once we blacklisted the client or deleted the client from the user-table,
> the websites were accessible again.
>
> Sid
>
> On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada 
> wrote:
>
>> With 8.6.0.9, no issues.
>>
>>
>>
>> (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25
>>
>> 35.186.224.25 172.16.122.193  6443   58612  0/0 024  3
>> tunnel 2306 a5   69 11747  17
>>
>> 172.16.126.14335.186.224.25   665364 4430/0 024  0
>> tunnel 1718 1a   29 3592   TC  26
>>
>> 172.18.91.115 35.186.224.25   656982 4430/0 00   0
>> tunnel 1102 505  14524120  C   29
>>
>> 172.16.174.33 35.186.224.25   654373 4430/0 024  0
>> tunnel 2773 6da  9576   1018764TC  21
>>
>> 35.186.224.25 172.16.166.198  6443   60052  0/0 024  1
>> tunnel 133  de   371269692 31
>>
>> 172.16.172.51 35.186.224.25   663940 4430/0 024  3
>> tunnel 862  5c   17 2849   TC  30
>>
>> 172.19.90.133 35.186.224.25   654371 4430/0 024  0
>> tunnel 1509 890  16133426  TC  18
>>
>> 172.19.91.45  35.186.224.25   662292 4430/0 024  2
>> tunnel 1630 4d   14 2502   TC  27
>>
>> 35.186.224.25 172.16.166.198  6443   60050  0/0 024  14
>> tunnel 133  de   24 8727   31
>>
>> 172.16.176.74 35.186.224.25   658973 4430/0 024  2
>> tunnel 1964 236  35 5322   TC  16
>>
>> 172.16.176.19335.186.224.25   661015 4430/0 024  1
>> tunnel 2160 10   44 15853  FTC 20
>>
>>
>>
>> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
>> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs
>> *Sent:* Tuesday, September 7, 2021 10:59 AM
>> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>> *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from
>> wireless network - Aruba
>>
>>
>>
>> CAUTION: This email originated from outside of the University. Do not
>> click links or open attachments unless you recognize the sender and know
>> the content is safe.
>>
>>
>>
>> Not seeing that issue here.  We are on 8.7.1.4
>>
>>
>>
>> (aruba-controller-1) #show datapath session | include 35.186.224.25
>> 35.186.224.25 138.236.104.67  6443   64918  0/0 01   1
>> tunnel 6347 3cc  30750335  15
>> 138.236.82.47 35.186.224.25   657491 4430/0 00   4
>> tunnel 5540 382  179117595 C   30
>> 35.186.224.25 138.236.248.10  6443   54342  0/0 01   1
>> tunnel 972  e20916359  23
>> 35.186.224.25 138.236.82.47   6443   57491  0/0 01   4
>> tunnel 5540 382  18945940  30
>> 138.236.104.6735.186.224.25   664918 4430/0 00   1
>> tunnel 6347 3cd  34538357  C   29
>> 35.186.224.25 138.236.232.120 6443   61505  0/0 01   0
>> tunnel 7052 c15149165  22
>> 138.236.250.8535.186.224.25   654833 4430/0 00   1
>> tunnel 2686 1a   57 16206  C   27
>> 35.186.224.25 138.236.251.120 6

Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Sidharth Nandury]

This is very helpful! Thank you. We are planning to implement the
validusers-acl like you mentioned and restrict clients to only the IPs that
we provide via DHCP. The description is exactly what we are seeing.

Christopher, would it be alright if we reached out to you if we have
questions? I would hate to re-invent the wheel.

Thank you, again.

Sid

On Tue, Sep 7, 2021 at 3:53 PM Johnson, Christopher 
wrote:

> Sid,
>
>
> We know from personal experience of running into this issue several years
> ago. Like David, we’ve instituted a few validuserACLs – (I actually use
> aliases for those subnets – so that I can re-use them in other places and
> to give a description of those valid ip addresses).
>
> After finding the offending device, was 99% positive it was malicious –
> but as I dived into the Rabbit Hole – discovered it was just a stupid
> malfunctioning device…a Roku Stick. I’ve also seen this behavior on other
> devices that make use of a “Router/IP Sharing” SSID such as “Roku’s Dorm
> Mode” or “Internet Sharing” with Windows.
>
> The Roku generates it’s own SSID “AP Mode” while connecting to our
> infrastructure SSID – it’s not bridged – but routed based on the fact that
> when you connect your phone or computer to the Roku’s SSID – your assigned
> a 192.168.X private IP Address. What I suspect happened in our scenario
> (I’ll use your 23.185.0.1 address for example).
>
> 1. Student Connected Roku to Guest SSID
>
> 2. Roku Prompted Student to use “Dorm Mode”
>
> 3. Student Connected to Roku with iPhone or Computer with a “home page” of
> our institution’s website.
>
> 4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 –
> Destination: 192.168.X.X – instead of sending it to the “private network”
> wifi interface  to the user’s iPhone or computer – it sent it out the
> “infrastructure network” interface – which based on how a “User” gets into
> the table à
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
> – and was assigned the guest unauthenticated policy – denying all traffic –
> cept icmps.
>
>
>
> I first started suspecting things weren’t as “simple” as they may be when
> I noticed Roku’s were “claiming” the IP Addresses of Google – what was
> funny was seeing the Controller prevent one Roku from entering the
> User-Table with a Google IP Address – *ONLY because another Roku* had
> already sourced a packet with Google’s IP Address.
>
>
>
> If you add a “any any any deny” with “LOG” option enabled – you can see
> ALL the invalid sessions that would have entered the user-table – including
> their destinations.
>
>
> I was only able to “partially replicate the behavior” – but it’s still a
> strong case.
>
> A few links down below:
>
>
> How the user gets into the user-table of the controller? -
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
> IP Address Leaking -
> https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e
>
>
> Some info from the ArubaOS Hardening Guide
>
> https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364
> - Page 26 and 27 goes into detail about “validuser” and
> “local-valid-users” – “local-valid-users” requires the controller to have
> an IP Address on that VLAN interface. There’s also the “Enforce DHCP”
> option in each AAA Aruba Profile – essentially a per SSID setting.
>
>
>
>
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c
>
>
> https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4
>
>
>
> http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/
> -à (BROKEN LINK Now ☹)
> Unfortunately the video link I had from commsolutions – they had
> presentation demonstrating this issue but it’s a broken link now –one of
> their customers for whatever reason had their guests manually enter the ip
> addresses onto their ipads – and someone flip-flopped the “IP Address” and
> the “Default Gateway”….started denying traffic for the default
> gateway….whoops!
>
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Mike Fitzgerald
> *Sent:* Tuesday, September 07, 2021 12:16 PM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> *Subject:* Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites
> inaccessible from wireless network - Aruba
>
>
>
> Some people who received this message don't often get email from
> fi...@brandeis.edu. Learn why this is important
> 
>
> *[This message came from an external source. If suspicious, report to
> ab...@ilstu.edu ] *
>
> Check your valid user table 

RE: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Johnson, Christopher]

Sid,

We know from personal experience of running into this issue several years ago. 
Like David, we’ve instituted a few validuserACLs – (I actually use aliases for 
those subnets – so that I can re-use them in other places and to give a 
description of those valid ip addresses).

After finding the offending device, was 99% positive it was malicious – but as 
I dived into the Rabbit Hole – discovered it was just a stupid malfunctioning 
device…a Roku Stick. I’ve also seen this behavior on other devices that make 
use of a “Router/IP Sharing” SSID such as “Roku’s Dorm Mode” or “Internet 
Sharing” with Windows.

The Roku generates it’s own SSID “AP Mode” while connecting to our 
infrastructure SSID – it’s not bridged – but routed based on the fact that when 
you connect your phone or computer to the Roku’s SSID – your assigned a 
192.168.X private IP Address. What I suspect happened in our scenario (I’ll use 
your 23.185.0.1 address for example).

1. Student Connected Roku to Guest SSID
2. Roku Prompted Student to use “Dorm Mode”
3. Student Connected to Roku with iPhone or Computer with a “home page” of our 
institution’s website.
4. The Roku “mis-routed” a single packet -- Source: 23.185.0.1 – Destination: 
192.168.X.X – instead of sending it to the “private network” wifi interface  to 
the user’s iPhone or computer – it sent it out the “infrastructure network” 
interface – which based on how a “User” gets into the table --> 
https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
 – and was assigned the guest unauthenticated policy – denying all traffic – 
cept icmps.

I first started suspecting things weren’t as “simple” as they may be when I 
noticed Roku’s were “claiming” the IP Addresses of Google – what was funny was 
seeing the Controller prevent one Roku from entering the User-Table with a 
Google IP Address – ONLY because another Roku had already sourced a packet with 
Google’s IP Address.

If you add a “any any any deny” with “LOG” option enabled – you can see ALL the 
invalid sessions that would have entered the user-table – including their 
destinations.

I was only able to “partially replicate the behavior” – but it’s still a strong 
case.

A few links down below:

How the user gets into the user-table of the controller? - 
https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=95f4108f-5927-4700-891c-89fd218d0d4e
IP Address Leaking - 
https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=39207#bm69bbf671-9e9b-4302-b11c-0965445bff7e

Some info from the ArubaOS Hardening Guide
https://community.arubanetworks.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=d9518fcc-d8f1-440b-8f5d-68522d3be364
- Page 26 and 27 goes into detail about “validuser” and “local-valid-users” – 
“local-valid-users” requires the controller to have an IP Address on that VLAN 
interface. There’s also the “Enforce DHCP” option in each AAA Aruba Profile – 
essentially a per SSID setting.

https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=de2277df-ff0e-41c1-9efb-643c0a04cf5c
https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=99300862-622e-4dd5-9af4-f2d745b49db4

http://www.commsolutions.com/2011/10/eliminate-duplicate-client-entries-in-your-aruba-controller-for-clients-with-more-than-one-ip-address-or-network-interface/
 ---> (BROKEN LINK Now ☹)
Unfortunately the video link I had from commsolutions – they had presentation 
demonstrating this issue but it’s a broken link now –one of their customers for 
whatever reason had their guests manually enter the ip addresses onto their 
ipads – and someone flip-flopped the “IP Address” and the “Default 
Gateway”….started denying traffic for the default gateway….whoops!

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Mike Fitzgerald
Sent: Tuesday, September 07, 2021 12:16 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible 
from wireless network - Aruba

Some people who received this message don't often get email from 
fi...@brandeis.edu. Learn why this is 
important
[This message came from an external source. If suspicious, report to 
ab...@ilstu.edu]
Check your valid user table config to make sure you only allow the IP ranges 
your DHCP server would give a wireless client.  Otherwise, you can end up with 
user table entries for destination IP's and then those IP's get policed by the 
controller as you were seeing.  Aruba default for that config used to allow any 
any, which is bad...

Mike


On Tue, Sep 7, 2021 at 12:04 PM Sidharth Nandury 
mailto:nandu...@denison.edu>> wrote:
So. sigh!

It seems like an end client either statically or for some unknown reason got 
assigned the IP address for these websites. The role that the client was 
assigned had a policy to "deny" 

RE: rough start of semester on 9800-80 WLCs

[Chad Sawyer]

Thanks for the info.  I'll look into the AP service pack.  We haven't done one 
of those yet so kind of curious to see it in action.

Yeah we had mixed results with the 500 APs in a site tag.  Some of the areas on 
campus were fine.  I think client count had something to do with provoking it.  
Our highest population areas were the ones that saw the most capwap timeouts.  
Just curious- how are you checking the number of APs and site tags assigned to 
a wncd process?

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Rios, Hector J
Sent: Tuesday, September 7, 2021 12:57 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Chad,

Sorry to hear about the issues you ran into. We also started the semester with 
9800-80s, but we chose to go with 17.3.4.

Things went well for most of the day on the first day of classes, except for a 
single controller crash after business hours. Cisco has identified this as a 
bug on the 17.3.X:
CSCvx71141 - CPU HOG in RRM Process.

You should contact TAC to get more details. They might also be able to provide 
a workaround, depending on your configuration.

We also ran into the bug below, but this was fixed with an AP service pack. 
Cool feature BTW, it actually works.
CSCvz08781
Symptom: AP2800/3800/4800/1560/IW6300/ESW6300 Firmware Radio Crash on 17.3.4 
while passing client traffic.

There is also an issue on 17.3.4 that is impacting 9120s. Cisco is working on a 
service pack for this as well. Don't have more details on this.

Thank you on the information regarding the wncd processes. We also followed the 
best practices, but we do have controllers that have a few wncd processes with 
a little over 500 APs. No issues so far, other than we have noticed in a few 
instances that even though we only have 8 custom site tags, some WLCs will 
assign two sitetags to a single wncd process.  We are working with TAC on this.

We also have a substantial number of 2700 series AP. We encountered no major 
issues during the upgrade process.

Finally, we have noticed that L3 roaming is not working on our 802.1X and PSK 
SSIDs. I wonder if anyone has run into this issue as well?.

Best,

Hector Rios
UT Austin




From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 9:21 AM
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.4.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.4.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 
guide,
 specifically on site tag sizing.  Although the guide says up to 500 APs can 
safely be assigned to a site tag, that was far from the truth in our 
experience.  Several TAC folks missed it and it took our rep escalating the 
issue to a senior wireless design person from Cisco to finally find it.  She 
advised breaking up our site tags so that they didn't exceed 250 APs, which 

Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Mike Fitzgerald]

Check your valid user table config to make sure you only allow the IP
ranges your DHCP server would give a wireless client.  Otherwise, you can
end up with user table entries for destination IP's and then those IP's get
policed by the controller as you were seeing.  Aruba default for that
config used to allow any any, which is bad...

Mike


On Tue, Sep 7, 2021 at 12:04 PM Sidharth Nandury 
wrote:

> So. sigh!
>
> It seems like an end client either statically or for some unknown reason
> got assigned the IP address for these websites. The role that the client
> was assigned had a policy to "deny" traffic to the internet (as per
> design). The part that we did not know was that when a client is going to a
> particular destination, the controllers look at the user table to see if
> there is an IP and a route available before even going to the role-based
> ACLs.
>
> Once we blacklisted the client or deleted the client from the user-table,
> the websites were accessible again.
>
> Sid
>
> On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada 
> wrote:
>
>> With 8.6.0.9, no issues.
>>
>>
>>
>> (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25
>>
>> 35.186.224.25 172.16.122.193  6443   58612  0/0 024  3
>> tunnel 2306 a5   69 11747  17
>>
>> 172.16.126.14335.186.224.25   665364 4430/0 024  0
>> tunnel 1718 1a   29 3592   TC  26
>>
>> 172.18.91.115 35.186.224.25   656982 4430/0 00   0
>> tunnel 1102 505  14524120  C   29
>>
>> 172.16.174.33 35.186.224.25   654373 4430/0 024  0
>> tunnel 2773 6da  9576   1018764TC  21
>>
>> 35.186.224.25 172.16.166.198  6443   60052  0/0 024  1
>> tunnel 133  de   371269692 31
>>
>> 172.16.172.51 35.186.224.25   663940 4430/0 024  3
>> tunnel 862  5c   17 2849   TC  30
>>
>> 172.19.90.133 35.186.224.25   654371 4430/0 024  0
>> tunnel 1509 890  16133426  TC  18
>>
>> 172.19.91.45  35.186.224.25   662292 4430/0 024  2
>> tunnel 1630 4d   14 2502   TC  27
>>
>> 35.186.224.25 172.16.166.198  6443   60050  0/0 024  14
>> tunnel 133  de   24 8727   31
>>
>> 172.16.176.74 35.186.224.25   658973 4430/0 024  2
>> tunnel 1964 236  35 5322   TC  16
>>
>> 172.16.176.19335.186.224.25   661015 4430/0 024  1
>> tunnel 2160 10   44 15853  FTC 20
>>
>>
>>
>> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
>> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs
>> *Sent:* Tuesday, September 7, 2021 10:59 AM
>> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
>> *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from
>> wireless network - Aruba
>>
>>
>>
>> CAUTION: This email originated from outside of the University. Do not
>> click links or open attachments unless you recognize the sender and know
>> the content is safe.
>>
>>
>>
>> Not seeing that issue here.  We are on 8.7.1.4
>>
>>
>>
>> (aruba-controller-1) #show datapath session | include 35.186.224.25
>> 35.186.224.25 138.236.104.67  6443   64918  0/0 01   1
>> tunnel 6347 3cc  30750335  15
>> 138.236.82.47 35.186.224.25   657491 4430/0 00   4
>> tunnel 5540 382  179117595 C   30
>> 35.186.224.25 138.236.248.10  6443   54342  0/0 01   1
>> tunnel 972  e20916359  23
>> 35.186.224.25 138.236.82.47   6443   57491  0/0 01   4
>> tunnel 5540 382  18945940  30
>> 138.236.104.6735.186.224.25   664918 4430/0 00   1
>> tunnel 6347 3cd  34538357  C   29
>> 35.186.224.25 138.236.232.120 6443   61505  0/0 01   0
>> tunnel 7052 c15149165  22
>> 138.236.250.8535.186.224.25   654833 4430/0 00   1
>> tunnel 2686 1a   57 16206  C   27
>> 35.186.224.25 138.236.251.120 6443   51735  0/0 01   1
>> tunnel 7060 829 3140   F   13
>> 138.236.250.8535.186.224.25   654834 4430/0 00   2
>> tunnel 2686 18   152179792 C   27
>>
>>
>>
>> --Dan
>>
>>
>>
>> On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury 
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> Since last Monday we have seen a couple of different websites being
>> blocked on our Aruba wireless controllers. Spotify has been one of the
>> sites, as well as all websites hosted on IP 23.185.0.1 (which is our main
>> institution website - denison.edu). We can confirm that this is being
>> blocked as we see the "D" (Deny) 

RE: rough start of semester on 9800-80 WLCs

[Rios, Hector J]

Chad,

Sorry to hear about the issues you ran into. We also started the semester with 
9800-80s, but we chose to go with 17.3.4.

Things went well for most of the day on the first day of classes, except for a 
single controller crash after business hours. Cisco has identified this as a 
bug on the 17.3.X:
CSCvx71141 - CPU HOG in RRM Process.

You should contact TAC to get more details. They might also be able to provide 
a workaround, depending on your configuration.

We also ran into the bug below, but this was fixed with an AP service pack. 
Cool feature BTW, it actually works.
CSCvz08781
Symptom: AP2800/3800/4800/1560/IW6300/ESW6300 Firmware Radio Crash on 17.3.4 
while passing client traffic.

There is also an issue on 17.3.4 that is impacting 9120s. Cisco is working on a 
service pack for this as well. Don't have more details on this.

Thank you on the information regarding the wncd processes. We also followed the 
best practices, but we do have controllers that have a few wncd processes with 
a little over 500 APs. No issues so far, other than we have noticed in a few 
instances that even though we only have 8 custom site tags, some WLCs will 
assign two sitetags to a single wncd process.  We are working with TAC on this.

We also have a substantial number of 2700 series AP. We encountered no major 
issues during the upgrade process.

Finally, we have noticed that L3 roaming is not working on our 802.1X and PSK 
SSIDs. I wonder if anyone has run into this issue as well?.

Best,

Hector Rios
UT Austin




From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 9:21 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.4.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.4.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 
guide,
 specifically on site tag sizing.  Although the guide says up to 500 APs can 
safely be assigned to a site tag, that was far from the truth in our 
experience.  Several TAC folks missed it and it took our rep escalating the 
issue to a senior wireless design person from Cisco to finally find it.  She 
advised breaking up our site tags so that they didn't exceed 250 APs, which 
instantly resolved the CAPWAP timeouts.


Fourth issue:
Apparently some of the 2702i APs don't handle code upgrades gracefully with the 
9800s.  Cisco made it sound like this was a common issue.  After upgrading from 
17.3.3 to 17.3.4, several 2700s on campus were showing "%CAPWAP-3-ERRORLOG: 
Certificate verification failed!" when attempting to establish CAPWAP with the 
controllers.  This was resolved by manually recovering the APs by pushing an 
image from the downloads page to them via TFTP.  Luckily we have a staff member 
who's pretty skilled at automating this type of stuff.  These were the commands:

SSH to the affected AP
enable
!
(enter password if there is one)
!
debug capwap console cli
!
archive download-sw /overwrite /force-reload tftp://(tftp server 
IP)/ap3g2-k9w8-tar.153-3.JPJ7.tar
!

The AP will automatically reload, establish capwap with the controller, 
download the proper image, reload, and re-join the controller successfully.


Chad Sawyer
Network Engineer
USF Information Technology 

Re: [WIRELESS-LAN] [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Sidharth Nandury]

So. sigh!

It seems like an end client either statically or for some unknown reason
got assigned the IP address for these websites. The role that the client
was assigned had a policy to "deny" traffic to the internet (as per
design). The part that we did not know was that when a client is going to a
particular destination, the controllers look at the user table to see if
there is an IP and a route available before even going to the role-based
ACLs.

Once we blacklisted the client or deleted the client from the user-table,
the websites were accessible again.

Sid

On Tue, Sep 7, 2021 at 11:29 AM Norman Mourtada 
wrote:

> With 8.6.0.9, no issues.
>
>
>
> (Aruba7220-MC-05) *#show datapath session | include 35.186.224.25
>
> 35.186.224.25 172.16.122.193  6443   58612  0/0 024  3
> tunnel 2306 a5   69 11747  17
>
> 172.16.126.14335.186.224.25   665364 4430/0 024  0
> tunnel 1718 1a   29 3592   TC  26
>
> 172.18.91.115 35.186.224.25   656982 4430/0 00   0
> tunnel 1102 505  14524120  C   29
>
> 172.16.174.33 35.186.224.25   654373 4430/0 024  0
> tunnel 2773 6da  9576   1018764TC  21
>
> 35.186.224.25 172.16.166.198  6443   60052  0/0 024  1
> tunnel 133  de   371269692 31
>
> 172.16.172.51 35.186.224.25   663940 4430/0 024  3
> tunnel 862  5c   17 2849   TC  30
>
> 172.19.90.133 35.186.224.25   654371 4430/0 024  0
> tunnel 1509 890  16133426  TC  18
>
> 172.19.91.45  35.186.224.25   662292 4430/0 024  2
> tunnel 1630 4d   14 2502   TC  27
>
> 35.186.224.25 172.16.166.198  6443   60050  0/0 024  14
> tunnel 133  de   24 8727   31
>
> 172.16.176.74 35.186.224.25   658973 4430/0 024  2
> tunnel 1964 236  35 5322   TC  16
>
> 172.16.176.19335.186.224.25   661015 4430/0 024  1
> tunnel 2160 10   44 15853  FTC 20
>
>
>
> *From:* The EDUCAUSE Wireless Issues Community Group Listserv <
> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Dan Oachs
> *Sent:* Tuesday, September 7, 2021 10:59 AM
> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
> *Subject:* [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from
> wireless network - Aruba
>
>
>
> CAUTION: This email originated from outside of the University. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Not seeing that issue here.  We are on 8.7.1.4
>
>
>
> (aruba-controller-1) #show datapath session | include 35.186.224.25
> 35.186.224.25 138.236.104.67  6443   64918  0/0 01   1
> tunnel 6347 3cc  30750335  15
> 138.236.82.47 35.186.224.25   657491 4430/0 00   4
> tunnel 5540 382  179117595 C   30
> 35.186.224.25 138.236.248.10  6443   54342  0/0 01   1
> tunnel 972  e20916359  23
> 35.186.224.25 138.236.82.47   6443   57491  0/0 01   4
> tunnel 5540 382  18945940  30
> 138.236.104.6735.186.224.25   664918 4430/0 00   1
> tunnel 6347 3cd  34538357  C   29
> 35.186.224.25 138.236.232.120 6443   61505  0/0 01   0
> tunnel 7052 c15149165  22
> 138.236.250.8535.186.224.25   654833 4430/0 00   1
> tunnel 2686 1a   57 16206  C   27
> 35.186.224.25 138.236.251.120 6443   51735  0/0 01   1
> tunnel 7060 829 3140   F   13
> 138.236.250.8535.186.224.25   654834 4430/0 00   2
> tunnel 2686 18   152179792 C   27
>
>
>
> --Dan
>
>
>
> On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury 
> wrote:
>
> Hi All,
>
>
>
> Since last Monday we have seen a couple of different websites being
> blocked on our Aruba wireless controllers. Spotify has been one of the
> sites, as well as all websites hosted on IP 23.185.0.1 (which is our main
> institution website - denison.edu). We can confirm that this is being
> blocked as we see the "D" (Deny) Flag on the wireless controller. Below is
> an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8
> controllers seeing this?
>
>
>
> (wlc-Thor) #show datapath session | include 35.186.224.25
>
> Source IP or MAC  Destination IP  Prot SPort DPort Cntr Prio ToS Age
> Destination TAge PacketsBytes  Flags   CPU ID
>
> - ---  - -   --- ---
> ---  -- -- --- ---
>
> 10.143.203.26 35.186.224.25 

RE: Cisco WLC 5508 software recommendations

[Glinsky, Eric]

So far so good with 8.5.171.0 on 8540s and a variety of APs.
Eric Glinsky
Network Administrator
University of Connecticut
ITS - Network Operations
Temporary Administration Building
25 Gampel Service Drive | Storrs, CT 06269-1138
(860) 486-9199
e...@uconn.edu


From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Entwistle, Bruce
Sent: Tuesday, September 7, 2021 11:43 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [WIRELESS-LAN] Cisco WLC 5508 software recommendations

*Message sent from a system outside of UConn.*


We are looking to upgrade our pair of 5508 controllers from the current 
version, 8.5.151.0.  We cannot move beyond the 8.5 code as we are still using 
some older 3500 access points. I have seen many comments regarding versions of 
code to avoid, but was looking to see what versions the group has found to be 
stable and would recommend moving to.



Thank you

Bruce Entwistle

Network Manager

University of Redlands






**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

Cisco WLC 5508 software recommendations

[Entwistle, Bruce]

We are looking to upgrade our pair of 5508 controllers from the current 
version, 8.5.151.0.  We cannot move beyond the 8.5 code as we are still using 
some older 3500 access points. I have seen many comments regarding versions of 
code to avoid, but was looking to see what versions the group has found to be 
stable and would recommend moving to.



Thank you

Bruce Entwistle

Network Manager

University of Redlands






**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Norman Mourtada]

With 8.6.0.9, no issues.

(Aruba7220-MC-05) *#show datapath session | include 35.186.224.25
35.186.224.25 172.16.122.193  6443   58612  0/0 024  3   tunnel 
2306 a5   69 11747  17
172.16.126.14335.186.224.25   665364 4430/0 024  0   tunnel 
1718 1a   29 3592   TC  26
172.18.91.115 35.186.224.25   656982 4430/0 00   0   tunnel 
1102 505  14524120  C   29
172.16.174.33 35.186.224.25   654373 4430/0 024  0   tunnel 
2773 6da  9576   1018764TC  21
35.186.224.25 172.16.166.198  6443   60052  0/0 024  1   tunnel 
133  de   371269692 31
172.16.172.51 35.186.224.25   663940 4430/0 024  3   tunnel 
862  5c   17 2849   TC  30
172.19.90.133 35.186.224.25   654371 4430/0 024  0   tunnel 
1509 890  16133426  TC  18
172.19.91.45  35.186.224.25   662292 4430/0 024  2   tunnel 
1630 4d   14 2502   TC  27
35.186.224.25 172.16.166.198  6443   60050  0/0 024  14  tunnel 
133  de   24 8727   31
172.16.176.74 35.186.224.25   658973 4430/0 024  2   tunnel 
1964 236  35 5322   TC  16
172.16.176.19335.186.224.25   661015 4430/0 024  1   tunnel 
2160 10   44 15853  FTC 20

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Dan Oachs
Sent: Tuesday, September 7, 2021 10:59 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [EXTERNAL] Re: [WIRELESS-LAN] Websites inaccessible from wireless 
network - Aruba

CAUTION: This email originated from outside of the University. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.

Not seeing that issue here.  We are on 8.7.1.4

(aruba-controller-1) #show datapath session | include 35.186.224.25
35.186.224.25 138.236.104.67  6443   64918  0/0 01   1   tunnel 
6347 3cc  30750335  15
138.236.82.47 35.186.224.25   657491 4430/0 00   4   tunnel 
5540 382  179117595 C   30
35.186.224.25 138.236.248.10  6443   54342  0/0 01   1   tunnel 
972  e20916359  23
35.186.224.25 138.236.82.47   6443   57491  0/0 01   4   tunnel 
5540 382  18945940  30
138.236.104.6735.186.224.25   664918 4430/0 00   1   tunnel 
6347 3cd  34538357  C   29
35.186.224.25 138.236.232.120 6443   61505  0/0 01   0   tunnel 
7052 c15149165  22
138.236.250.8535.186.224.25   654833 4430/0 00   1   tunnel 
2686 1a   57 16206  C   27
35.186.224.25 138.236.251.120 6443   51735  0/0 01   1   tunnel 
7060 829 3140   F   13
138.236.250.8535.186.224.25   654834 4430/0 00   2   tunnel 
2686 18   152179792 C   27

--Dan

On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury 
mailto:nandu...@denison.edu>> wrote:
Hi All,

Since last Monday we have seen a couple of different websites being blocked on 
our Aruba wireless controllers. Spotify has been one of the sites, as well as 
all websites hosted on IP 23.185.0.1 (which is our main institution website - 
denison.edu). We can confirm that this is being blocked as 
we see the "D" (Deny) Flag on the wireless controller. Below is an example of 
traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 controllers 
seeing this?


(wlc-Thor) #show datapath session | include 35.186.224.25

Source IP or MAC  Destination IP  Prot SPort DPort Cntr Prio ToS Age 
Destination TAge PacketsBytes  Flags   CPU ID

- ---  - -   --- --- 
---  -- -- --- ---

10.143.203.26 35.186.224.25   652082 4430/0 00   0   tunnel 
640  10  0  FDYCA   21

10.143.195.85 35.186.224.25   659767 4430/0 00   0   tunnel 
5357 00  0  FDYCA   27

10.143.225.17835.186.224.25   652292 4430/0 00   0   tunnel 
6753 10  0  FDYCA   19

10.143.195.85 35.186.224.25   659766 4430/0 00   0   tunnel 
5357 10  0  FDYCA   27



(wlc-Thor) #show datapath session | include 23.185.0.1
10.143.228.16 23.185.0.1  659500 4430/0 00   0   tunnel 
16789 a0  0  FDYCA   18
10.143.244.15123.185.0.1  658758 4430/0 00   0   

Re: [WIRELESS-LAN] Websites inaccessible from wireless network - Aruba

[Dan Oachs]

Not seeing that issue here.  We are on 8.7.1.4

(aruba-controller-1) #show datapath session | include 35.186.224.25
35.186.224.25 138.236.104.67  6443   64918  0/0 01   1
tunnel 6347 3cc  30750335  15
138.236.82.47 35.186.224.25   657491 4430/0 00   4
tunnel 5540 382  179117595 C   30
35.186.224.25 138.236.248.10  6443   54342  0/0 01   1
tunnel 972  e20916359  23
35.186.224.25 138.236.82.47   6443   57491  0/0 01   4
tunnel 5540 382  18945940  30
138.236.104.6735.186.224.25   664918 4430/0 00   1
tunnel 6347 3cd  34538357  C   29
35.186.224.25 138.236.232.120 6443   61505  0/0 01   0
tunnel 7052 c15149165  22
138.236.250.8535.186.224.25   654833 4430/0 00   1
tunnel 2686 1a   57 16206  C   27
35.186.224.25 138.236.251.120 6443   51735  0/0 01   1
tunnel 7060 829 3140   F   13
138.236.250.8535.186.224.25   654834 4430/0 00   2
tunnel 2686 18   152179792 C   27

--Dan

On Tue, Sep 7, 2021 at 9:40 AM Sidharth Nandury 
wrote:

> Hi All,
>
> Since last Monday we have seen a couple of different websites being
> blocked on our Aruba wireless controllers. Spotify has been one of the
> sites, as well as all websites hosted on IP 23.185.0.1 (which is our main
> institution website - denison.edu). We can confirm that this is being
> blocked as we see the "D" (Deny) Flag on the wireless controller. Below is
> an example of traffic being blocked to Spotify. Is anyone suing Aruba AOS 8
> controllers seeing this?
>
> (wlc-Thor) #show datapath session | include 35.186.224.25
>
> Source IP or MAC  Destination IP  Prot SPort DPort Cntr Prio ToS Age
> Destination TAge PacketsBytes  Flags   CPU ID
>
> - ---  - -   --- ---
> ---  -- -- --- ---
>
> 10.143.203.26 35.186.224.25   652082 4430/0 00   0
> tunnel 640  10  0  *FDYCA *  21
>
> 10.143.195.85 35.186.224.25   659767 4430/0 00   0
> tunnel 5357 00  0*  FDYCA*   27
>
> 10.143.225.17835.186.224.25   652292 4430/0 00   0
> tunnel 6753 10  0 * FDYCA *  19
>
> 10.143.195.85 35.186.224.25   659766 4430/0 00   0
> tunnel 5357 10  0  *FDYCA *  27
>
>
> (wlc-Thor) #show datapath session | include 23.185.0.1
> 10.143.228.16 23.185.0.1  659500 4430/0 00   0
> tunnel 16789 a0  0  *FDYCA*   18
> 10.143.244.15123.185.0.1  658758 4430/0 00   0
> tunnel 553  10  0  *FDYCA*   23
> 10.143.228.24723.185.0.1  659063 4430/0 00   0
> tunnel 13188 a6  384*FDYCA*   27
> 10.143.228.24723.185.0.1  659062 4430/0 00   0
> tunnel 13188 a6  384*FDYCA*   27
> 10.143.196.26 23.185.0.1  650851 4430/0 00   0
> tunnel 5631 10  0  *FDYCA*   17
> 10.143.196.26 23.185.0.1  650852 4430/0 00   0
> tunnel 5631 10  0  *FDYCA*   17
> 10.143.196.26 23.185.0.1  650853 4430/0 00   0
> tunnel 5631 10  0  *FDYCA*   17
>
>
> We have two 7240xm controllers running AOS v8.6.9 in a cluster with a
> Mobility Conductor as a VM. We have a ticket open with TAC and have
> escalated it up to ERT, but wanted to also reach out to others.
>
>
> Thank you.
>
> Sid
>
>
> --
>
> [image: Denison University] 
>
> *Sidharth S. Nandury*
> (He, Him, His)
> *Infrastructure and Operations Manager*
> Information Technology Services
>
> 100 West College Street, Granville, OH 43023  | 
> Burton
> Hall 
> Office: 740-587-5533 <1-740-587-5533> | Mobile: 516-314-4413
> <1-516-314-4413>
> nand...@denison.edu
> https://denison.edu/campus/technology/service-desk
>
> NOTICE: This email message and all attachments transmitted with it may
> contain legally privileged and confidential information intended solely for
> the use of the addressee. If the reader of this message is not the intended
> recipient, you are hereby notified that any reading, dissemination,
> distribution, copying, or other use of this message or its attachments is
> strictly prohibited. If you have received this message in error, please
> notify the sender immediately by phone or by email, and delete this message
> and all copies and backups thereof.
>
> 

RE: rough start of semester on 9800-80 WLCs

[Chad Sawyer]

Oh yes good catch!  Yes it should be 17.3.4.

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Sullivan, Don
Sent: Tuesday, September 7, 2021 10:37 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just to clarify - is that 17.4.4 or 17.3.4?

Don Sullivan
Network Administrator
Technology Services

205-726-2111 | office
dsulli...@samford.edu
LinkedIn
www.samford.edu
800 Lakeshore Drive
Birmingham, AL 
35229

[Samford Samford University Logo]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 09:21
To: 
WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [EXTERNAL][WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.3.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.3.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 
guide,
 specifically on site tag sizing.  Although the guide says up to 500 APs can 
safely be assigned to a site tag, that was far from the truth in our 
experience.  Several TAC folks missed it and it took our rep escalating the 
issue to a senior wireless design person from Cisco to finally find it.  She 
advised breaking up our site tags so that they didn't exceed 250 APs, which 
instantly resolved the CAPWAP timeouts.


Fourth issue:
Apparently some of the 2702i APs don't handle code upgrades gracefully with the 
9800s.  Cisco made it sound like this was a common issue.  After upgrading from 
17.3.3 to 17.3.4, several 2700s on campus were showing "%CAPWAP-3-ERRORLOG: 
Certificate verification failed!" when attempting to establish CAPWAP with the 
controllers.  This was resolved by manually recovering the APs by pushing an 
image from the downloads page 

Websites inaccessible from wireless network - Aruba

[Sidharth Nandury]

Hi All,

Since last Monday we have seen a couple of different websites being blocked
on our Aruba wireless controllers. Spotify has been one of the sites, as
well as all websites hosted on IP 23.185.0.1 (which is our main institution
website - denison.edu). We can confirm that this is being blocked as we see
the "D" (Deny) Flag on the wireless controller. Below is an example of
traffic being blocked to Spotify. Is anyone suing Aruba AOS 8 controllers
seeing this?

(wlc-Thor) #show datapath session | include 35.186.224.25

Source IP or MAC  Destination IP  Prot SPort DPort Cntr Prio ToS Age
Destination TAge PacketsBytes  Flags   CPU ID

- ---  - -   --- ---
---  -- -- --- ---

10.143.203.26 35.186.224.25   652082 4430/0 00   0
tunnel 640  10  0  *FDYCA *  21

10.143.195.85 35.186.224.25   659767 4430/0 00   0
tunnel 5357 00  0*  FDYCA*   27

10.143.225.17835.186.224.25   652292 4430/0 00   0
tunnel 6753 10  0 * FDYCA *  19

10.143.195.85 35.186.224.25   659766 4430/0 00   0
tunnel 5357 10  0  *FDYCA *  27


(wlc-Thor) #show datapath session | include 23.185.0.1
10.143.228.16 23.185.0.1  659500 4430/0 00   0
tunnel 16789 a0  0  *FDYCA*   18
10.143.244.15123.185.0.1  658758 4430/0 00   0
tunnel 553  10  0  *FDYCA*   23
10.143.228.24723.185.0.1  659063 4430/0 00   0
tunnel 13188 a6  384*FDYCA*   27
10.143.228.24723.185.0.1  659062 4430/0 00   0
tunnel 13188 a6  384*FDYCA*   27
10.143.196.26 23.185.0.1  650851 4430/0 00   0
tunnel 5631 10  0  *FDYCA*   17
10.143.196.26 23.185.0.1  650852 4430/0 00   0
tunnel 5631 10  0  *FDYCA*   17
10.143.196.26 23.185.0.1  650853 4430/0 00   0
tunnel 5631 10  0  *FDYCA*   17


We have two 7240xm controllers running AOS v8.6.9 in a cluster with a
Mobility Conductor as a VM. We have a ticket open with TAC and have
escalated it up to ERT, but wanted to also reach out to others.


Thank you.

Sid


-- 

[image: Denison University] 

*Sidharth S. Nandury*
(He, Him, His)
*Infrastructure and Operations Manager*
Information Technology Services

100 West College Street, Granville, OH 43023
 | Burton
Hall 
Office: 740-587-5533 <1-740-587-5533> | Mobile: 516-314-4413
<1-516-314-4413>
nand...@denison.edu
https://denison.edu/campus/technology/service-desk

NOTICE: This email message and all attachments transmitted with it may
contain legally privileged and confidential information intended solely for
the use of the addressee. If the reader of this message is not the intended
recipient, you are hereby notified that any reading, dissemination,
distribution, copying, or other use of this message or its attachments is
strictly prohibited. If you have received this message in error, please
notify the sender immediately by phone or by email, and delete this message
and all copies and backups thereof.

*Please consider the environment before printing this email.*

**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community

RE: rough start of semester on 9800-80 WLCs

[Sullivan, Don]

Just to clarify - is that 17.4.4 or 17.3.4?

Don Sullivan
Network Administrator
Technology Services

205-726-2111 | office
dsulli...@samford.edu
LinkedIn
www.samford.edu
800 Lakeshore Drive
Birmingham, AL 
35229

[Samford Samford University Logo]

From: The EDUCAUSE Wireless Issues Community Group Listserv 
 On Behalf Of Chad Sawyer
Sent: Tuesday, September 7, 2021 09:21
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: [EXTERNAL][WIRELESS-LAN] rough start of semester on 9800-80 WLCs

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.4.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.4.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 
guide,
 specifically on site tag sizing.  Although the guide says up to 500 APs can 
safely be assigned to a site tag, that was far from the truth in our 
experience.  Several TAC folks missed it and it took our rep escalating the 
issue to a senior wireless design person from Cisco to finally find it.  She 
advised breaking up our site tags so that they didn't exceed 250 APs, which 
instantly resolved the CAPWAP timeouts.


Fourth issue:
Apparently some of the 2702i APs don't handle code upgrades gracefully with the 
9800s.  Cisco made it sound like this was a common issue.  After upgrading from 
17.3.3 to 17.3.4, several 2700s on campus were showing "%CAPWAP-3-ERRORLOG: 
Certificate verification failed!" when attempting to establish CAPWAP with the 
controllers.  This was resolved by manually recovering the APs by pushing an 
image from the downloads page to them via TFTP.  Luckily we have a staff member 
who's pretty skilled at automating this type of stuff.  These were the commands:

SSH to the affected AP
enable
!
(enter password if there is one)
!
debug capwap console cli
!
archive download-sw /overwrite /force-reload tftp://(tftp server 
IP)/ap3g2-k9w8-tar.153-3.JPJ7.tar
!

The AP will automatically reload, establish capwap with the controller, 
download the proper image, reload, and re-join the controller successfully.


Chad Sawyer
Network Engineer
USF Information Technology 
www.usf.edu/it
13220 USF Laurel Dr, MDF 2128, Tampa, FL 33620
O: 813-974-1342
E: chadsaw...@usf.edu
[https://www.usf.edu/images/ucm/marketing/logos/email-sigs/email-signature-bull-u-usf-preem-240x68.png]


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at 
https://www.educause.edu/community

rough start of semester on 9800-80 WLCs

[Chad Sawyer]

Just sending a heads up in case anyone else hits these.  This was our first 
semester with a full campus since moving everything over to our 9800-80 pairs.  
They've been in production for much of the past 12 months and the performance 
was fine when campus was empty.  Under load was another story.

First issue:
Code 17.3.3 has the following bugs that were causing frequent HA failovers that 
reference the wncd process.  This was resolved by upgrading to 17.4.4.
CSCvx37499- Controller reloads with the reason "Critical process wncd fault on 
rp_0_0 (rc=139)
CSCvy20300- Primary controller in HA frequently ends abnormally

Second issue:
Unfortunately these failovers also provoked one of the units to lose the 
contents of its bootflash and get stuck in rommon mode, so we had to recover it 
via the booting to USB routine.  This was also due to a 17.3.3 bug and has been 
hopefully resolved so far by upgrading to 17.4.4.
CSCvy73836- C9800-80 controller goes to rommon after multiple failovers due to 
power cycling

Third issue:
The nastiest thing though was unrelated to bugs.  It was CAPWAP timeouts that 
only occurred in busy areas of campus.  AP uptime would show months, but CAPWAP 
uptimes were constantly resetting to zero.  The logs on the AP would show the 
following message: "Going to restart CAPWAP (reason : data keepalive not 
received)"  We wasted a lot of time troubleshooting this as a connectivity 
issue between our APs and controller, but that wasn't the cause.

This problem was a result of our following Cisco's 9800 best practice 
guide,
 specifically on site tag sizing.  Although the guide says up to 500 APs can 
safely be assigned to a site tag, that was far from the truth in our 
experience.  Several TAC folks missed it and it took our rep escalating the 
issue to a senior wireless design person from Cisco to finally find it.  She 
advised breaking up our site tags so that they didn't exceed 250 APs, which 
instantly resolved the CAPWAP timeouts.


Fourth issue:
Apparently some of the 2702i APs don't handle code upgrades gracefully with the 
9800s.  Cisco made it sound like this was a common issue.  After upgrading from 
17.3.3 to 17.3.4, several 2700s on campus were showing "%CAPWAP-3-ERRORLOG: 
Certificate verification failed!" when attempting to establish CAPWAP with the 
controllers.  This was resolved by manually recovering the APs by pushing an 
image from the downloads page to them via TFTP.  Luckily we have a staff member 
who's pretty skilled at automating this type of stuff.  These were the commands:

SSH to the affected AP
enable
!
(enter password if there is one)
!
debug capwap console cli
!
archive download-sw /overwrite /force-reload tftp://(tftp server 
IP)/ap3g2-k9w8-tar.153-3.JPJ7.tar
!

The AP will automatically reload, establish capwap with the controller, 
download the proper image, reload, and re-join the controller successfully.


Chad Sawyer
Network Engineer
USF Information Technology 
www.usf.edu/it
13220 USF Laurel Dr, MDF 2128, Tampa, FL 33620
O: 813-974-1342
E: chadsaw...@usf.edu
[https://www.usf.edu/images/ucm/marketing/logos/email-sigs/email-signature-bull-u-usf-preem-240x68.png]


**
Replies to EDUCAUSE Community Group emails are sent to the entire community 
list. If you want to reply only to the person who sent the message, copy and 
paste their email address and forward the email reply. Additional participation 
and subscription information can be found at https://www.educause.edu/community