RE: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller losses

2018-01-05 Thread Turner, Ryan H 
In a perfect world…  We can likely do this, but our network design is a lot 
flatter.  However, there are opportunities to carve this up a bit and mitigate 
it.

From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Fredrik L. Andersen
Sent: Friday, January 5, 2018 1:36 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

Hi,

Agree with you both with better network design, controllers and AP should not 
be on same L2. Use DNS for MC discovery.

You should also check out NG architecture for AOS8 with clustering for HA.

Best regards


Fredrik L. Andersen
+ 47 930 888 15


Sendt fra min iPhone

5. jan. 2018 kl. 19:25 skrev Norton, Thomas (Network Operations) 
<tnort...@liberty.edu<mailto:tnort...@liberty.edu>>:
Hey Ryan,

I agree with Amel, I highly recommend breaking out your aps separate from your 
controller management VLAN and utilizing DHCP for discovery.

We break out our ap management VLANs from our controller management VLAN and 
have the ap VLANs broken up into multiple geographic VTP domains to mitigate 
this.

With that said we have had our own set of challenges from an HA perspective, as 
we have had to tune our ha heartbeat timers, and configuration to meet our 
needs…

-T.J.
Liberty University


From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of Amel Caldwell <am...@uw.edu<mailto:am...@uw.edu>>
Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Date: Friday, January 5, 2018 at 12:42 PM
To: 
"WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

Hi Ryan—

We have a similar setup, our main campus has around 7,000 APs with one master 
controller.  We have separate AP management VLANs in each of our buildings (we 
don’t span VLANs across multiple buildings here) and use DHCP options for 
master controller discovery.  We still get a ton on pings looking for a lost 
controller but the infrastructure handles the pings better than they do ARPs.  
It may help if you separate the controller management and AP management onto 
separate VLANs and use DHCP options; this would have the effect of changing the 
ARP to ICMP traffic and hopefully that would be enough to weather the event of 
a lost controller.

I do wholeheartedly agree that Aruba implenting a back-off mechanism to lessen 
this impact over time would be great.  I am also not real happy with how Aruba 
implemented the “heartbeat” option for the standby-controller to verify the 
primary is still up and it really does not scale well.

Amel Caldwell
University of Washington UW-IT
Wi-Fi Network Engineer
Wi-Fi Service Manager

am...@uw.edu<mailto:am...@uw.edu>
206-543-2915



From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> 
on behalf of "Turner, Ryan H" 
<rhtur...@email.unc.edu<mailto:rhtur...@email.unc.edu>>
Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Date: Friday, January 5, 2018 at 9:14 AM
To: 
"WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>" 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>>
Subject: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

All:

Based on design recommendations from Aruba, our 10,000 AP network has been 
broken up into a few management domains.  For example, Main Campus has 
approximately 5,000 access points, and the controllers and access points share 
the same VLAN.

What we have noticed is that if we lose a controller (or shut it down for 
maintenance or a move), the access points start ARPing like crazy for the 
downed controller.  We can see in excess of 1,000 ARPs a second in the 
management VLAN.  This has the negative side effect of causing CPU spikes 
across certain models of switches on campus, and we lose management to those 
switches.  User traffic doesn’t generally seem affected, but SNMP monitoring 
ceases.  We are wondering if others have seen this, or designed around 
mitigating this.  This is definitely a scaling issue, and we feel as though 
Aruba could develop back-off mechanisms from allowing High Availability to 
essentially DoS parts of campus with ARP.

Thanks!

Ryan Turner
Manager of Network Operations
ITS Communication Technologie

Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller losses

2018-01-05 Thread Fredrik L. Andersen 
Hi,

Agree with you both with better network design, controllers and AP should not 
be on same L2. Use DNS for MC discovery. 

You should also check out NG architecture for AOS8 with clustering for HA.

Best regards

Fredrik L. Andersen
+ 47 930 888 15

Sendt fra min iPhone

> 5. jan. 2018 kl. 19:25 skrev Norton, Thomas (Network Operations) 
> <tnort...@liberty.edu>:
> 
> Hey Ryan,
>  
> I agree with Amel, I highly recommend breaking out your aps separate from 
> your controller management VLAN and utilizing DHCP for discovery.
>  
> We break out our ap management VLANs from our controller management VLAN and 
> have the ap VLANs broken up into multiple geographic VTP domains to mitigate 
> this.
>  
> With that said we have had our own set of challenges from an HA perspective, 
> as we have had to tune our ha heartbeat timers, and configuration to meet our 
> needs…
>  
> -T.J.
> Liberty University
>  
>  
> From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
> <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Amel Caldwell <am...@uw.edu>
> Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
> <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Date: Friday, January 5, 2018 at 12:42 PM
> To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Subject: Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during 
> controller losses
>  
> Hi Ryan—
>  
> We have a similar setup, our main campus has around 7,000 APs with one master 
> controller.  We have separate AP management VLANs in each of our buildings 
> (we don’t span VLANs across multiple buildings here) and use DHCP options for 
> master controller discovery.  We still get a ton on pings looking for a lost 
> controller but the infrastructure handles the pings better than they do ARPs. 
>  It may help if you separate the controller management and AP management onto 
> separate VLANs and use DHCP options; this would have the effect of changing 
> the ARP to ICMP traffic and hopefully that would be enough to weather the 
> event of a lost controller.
>  
> I do wholeheartedly agree that Aruba implenting a back-off mechanism to 
> lessen this impact over time would be great.  I am also not real happy with 
> how Aruba implemented the “heartbeat” option for the standby-controller to 
> verify the primary is still up and it really does not scale well.
>  
> Amel Caldwell
> University of Washington UW-IT
> Wi-Fi Network Engineer
> Wi-Fi Service Manager
>  
> am...@uw.edu
> 206-543-2915
>  
>  
>  
> From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
> <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of "Turner, Ryan H" 
> <rhtur...@email.unc.edu>
> Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
> <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Date: Friday, January 5, 2018 at 9:14 AM
> To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
> Subject: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
> losses
>  
> All:
>  
> Based on design recommendations from Aruba, our 10,000 AP network has been 
> broken up into a few management domains.  For example, Main Campus has 
> approximately 5,000 access points, and the controllers and access points 
> share the same VLAN.
>  
> What we have noticed is that if we lose a controller (or shut it down for 
> maintenance or a move), the access points start ARPing like crazy for the 
> downed controller.  We can see in excess of 1,000 ARPs a second in the 
> management VLAN.  This has the negative side effect of causing CPU spikes 
> across certain models of switches on campus, and we lose management to those 
> switches.  User traffic doesn’t generally seem affected, but SNMP monitoring 
> ceases.  We are wondering if others have seen this, or designed around 
> mitigating this.  This is definitely a scaling issue, and we feel as though 
> Aruba could develop back-off mechanisms from allowing High Availability to 
> essentially DoS parts of campus with ARP.
>  
> Thanks!
>  
> Ryan Turner
> Manager of Network Operations
> ITS Communication Technologies
> The University of North Carolina at Chapel Hill
>  
> r...@unc.edu
> +1 919 445 0113 Office
> +1 919 274 7926 Mobile
>  
> ** Participation and subscription information for this EDUCAUSE 
> Constituent Group discussion list can be found at 
> http://www.educause.edu/discuss.
> ** Participation and subscription information for this EDUCAUSE 
> Constituent Group discussion list can be found at 
> http://www.educause.edu/discuss.
> ** Participation and subscription information for this EDUCAUSE 
> Constituent Group discussion list can be found at 
> http://www.educause.edu/discuss.

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/discuss.

Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller losses

2018-01-05 Thread Norton, Thomas (Network Operations) 
Hey Ryan,

I agree with Amel, I highly recommend breaking out your aps separate from your 
controller management VLAN and utilizing DHCP for discovery.

We break out our ap management VLANs from our controller management VLAN and 
have the ap VLANs broken up into multiple geographic VTP domains to mitigate 
this.

With that said we have had our own set of challenges from an HA perspective, as 
we have had to tune our ha heartbeat timers, and configuration to meet our 
needs…

-T.J.
Liberty University


From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of Amel Caldwell <am...@uw.edu>
Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Date: Friday, January 5, 2018 at 12:42 PM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

Hi Ryan—

We have a similar setup, our main campus has around 7,000 APs with one master 
controller.  We have separate AP management VLANs in each of our buildings (we 
don’t span VLANs across multiple buildings here) and use DHCP options for 
master controller discovery.  We still get a ton on pings looking for a lost 
controller but the infrastructure handles the pings better than they do ARPs.  
It may help if you separate the controller management and AP management onto 
separate VLANs and use DHCP options; this would have the effect of changing the 
ARP to ICMP traffic and hopefully that would be enough to weather the event of 
a lost controller.

I do wholeheartedly agree that Aruba implenting a back-off mechanism to lessen 
this impact over time would be great.  I am also not real happy with how Aruba 
implemented the “heartbeat” option for the standby-controller to verify the 
primary is still up and it really does not scale well.

Amel Caldwell
University of Washington UW-IT
Wi-Fi Network Engineer
Wi-Fi Service Manager

am...@uw.edu<mailto:am...@uw.edu>
206-543-2915



From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of "Turner, Ryan H" 
<rhtur...@email.unc.edu>
Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Date: Friday, January 5, 2018 at 9:14 AM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

All:

Based on design recommendations from Aruba, our 10,000 AP network has been 
broken up into a few management domains.  For example, Main Campus has 
approximately 5,000 access points, and the controllers and access points share 
the same VLAN.

What we have noticed is that if we lose a controller (or shut it down for 
maintenance or a move), the access points start ARPing like crazy for the 
downed controller.  We can see in excess of 1,000 ARPs a second in the 
management VLAN.  This has the negative side effect of causing CPU spikes 
across certain models of switches on campus, and we lose management to those 
switches.  User traffic doesn’t generally seem affected, but SNMP monitoring 
ceases.  We are wondering if others have seen this, or designed around 
mitigating this.  This is definitely a scaling issue, and we feel as though 
Aruba could develop back-off mechanisms from allowing High Availability to 
essentially DoS parts of campus with ARP.

Thanks!

Ryan Turner
Manager of Network Operations
ITS Communication Technologies
The University of North Carolina at Chapel Hill

r...@unc.edu<mailto:r...@unc.edu>
+1 919 445 0113 Office
+1 919 274 7926 Mobile

** Participation and subscription information for this EDUCAUSE 
Constituent Group discussion list can be found at 
http://www.educause.edu/discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.educause.edu%2Fdiscuss=02%7C01%7Ctnorton7%40liberty.edu%7Cd1dd56c7c2f84531b2c508d55463a74a%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C636507709614711947=5rWQhHChbeQpUIubPVJZPb735BdhjqRHcZ%2FxPwQ0ziQ%3D=0>.
** Participation and subscription information for this EDUCAUSE 
Constituent Group discussion list can be found at 
http://www.educause.edu/discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.educause.edu%2Fdiscuss=02%7C01%7Ctnorton7%40liberty.edu%7Cd1dd56c7c2f84531b2c508d55463a74a%7Cbaf8218eb3024465a9934a39c97251b2%7C0%7C0%7C636507709614711947=5rWQhHChbeQpUIubPVJZPb735BdhjqRHcZ%2FxPwQ0ziQ%3D=0>.

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/discuss.

Re: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller losses

2018-01-05 Thread Amel Caldwell 
Hi Ryan—

We have a similar setup, our main campus has around 7,000 APs with one master 
controller.  We have separate AP management VLANs in each of our buildings (we 
don’t span VLANs across multiple buildings here) and use DHCP options for 
master controller discovery.  We still get a ton on pings looking for a lost 
controller but the infrastructure handles the pings better than they do ARPs.  
It may help if you separate the controller management and AP management onto 
separate VLANs and use DHCP options; this would have the effect of changing the 
ARP to ICMP traffic and hopefully that would be enough to weather the event of 
a lost controller.

I do wholeheartedly agree that Aruba implenting a back-off mechanism to lessen 
this impact over time would be great.  I am also not real happy with how Aruba 
implemented the “heartbeat” option for the standby-controller to verify the 
primary is still up and it really does not scale well.

Amel Caldwell
University of Washington UW-IT
Wi-Fi Network Engineer
Wi-Fi Service Manager

am...@uw.edu<mailto:am...@uw.edu>
206-543-2915



From: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> on behalf of "Turner, Ryan H" 
<rhtur...@email.unc.edu>
Reply-To: The EDUCAUSE Wireless Issues Constituent Group Listserv 
<WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Date: Friday, January 5, 2018 at 9:14 AM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: [WIRELESS-LAN] Aruba / HA / And ARP broadcasting during controller 
losses

All:

Based on design recommendations from Aruba, our 10,000 AP network has been 
broken up into a few management domains.  For example, Main Campus has 
approximately 5,000 access points, and the controllers and access points share 
the same VLAN.

What we have noticed is that if we lose a controller (or shut it down for 
maintenance or a move), the access points start ARPing like crazy for the 
downed controller.  We can see in excess of 1,000 ARPs a second in the 
management VLAN.  This has the negative side effect of causing CPU spikes 
across certain models of switches on campus, and we lose management to those 
switches.  User traffic doesn’t generally seem affected, but SNMP monitoring 
ceases.  We are wondering if others have seen this, or designed around 
mitigating this.  This is definitely a scaling issue, and we feel as though 
Aruba could develop back-off mechanisms from allowing High Availability to 
essentially DoS parts of campus with ARP.

Thanks!

Ryan Turner
Manager of Network Operations
ITS Communication Technologies
The University of North Carolina at Chapel Hill

r...@unc.edu<mailto:r...@unc.edu>
+1 919 445 0113 Office
+1 919 274 7926 Mobile

** Participation and subscription information for this EDUCAUSE 
Constituent Group discussion list can be found at 
http://www.educause.edu/discuss.

**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/discuss.

Aruba / HA / And ARP broadcasting during controller losses

2018-01-05 Thread Turner, Ryan H 
All:

Based on design recommendations from Aruba, our 10,000 AP network has been 
broken up into a few management domains.  For example, Main Campus has 
approximately 5,000 access points, and the controllers and access points share 
the same VLAN.

What we have noticed is that if we lose a controller (or shut it down for 
maintenance or a move), the access points start ARPing like crazy for the 
downed controller.  We can see in excess of 1,000 ARPs a second in the 
management VLAN.  This has the negative side effect of causing CPU spikes 
across certain models of switches on campus, and we lose management to those 
switches.  User traffic doesn't generally seem affected, but SNMP monitoring 
ceases.  We are wondering if others have seen this, or designed around 
mitigating this.  This is definitely a scaling issue, and we feel as though 
Aruba could develop back-off mechanisms from allowing High Availability to 
essentially DoS parts of campus with ARP.

Thanks!

Ryan Turner
Manager of Network Operations
ITS Communication Technologies
The University of North Carolina at Chapel Hill

r...@unc.edu
+1 919 445 0113 Office
+1 919 274 7926 Mobile


**
Participation and subscription information for this EDUCAUSE Constituent Group 
discussion list can be found at http://www.educause.edu/discuss.