RE: [WIRELESS-LAN] Aruba 8.7 issues
Chris, Well, I have a mix of problems. Since I moved to a MM dual controller scenario, roaming for students stopped working properly at 8.7.1.0. So I upgraded to 8.7.1.3 and that seemed to mostly fix it though there are some delays when going to a new ip address segment. Those delays didn’t exist for me in 6.8.x.x but the delays are only 30 to 60 seconds so livable for the moment. I didn’t have these delays when I was on a single controller with the same configuration. I will fix that this summer when I make the IP’s follow everyone from zone to zone which should get rid of this roaming issue. But lately I have had a few complaints where they say wireless went away on them. I don’t see evidence in Airwave. I have only been on 8.7.1.3 for about 2 weeks now so these complaints make me paranoid. Some of these complaints are from our own tech team. When I go in to test, I don’t observe it. I am hopefully going to meet with one of them today to see why they keep having issues with what they think is poor signal quality when it is perfect for me in the same location. Before I moved to Mobility Manager, I had maybe 2 wireless complaints a semester. After moving from 6.x.x.x to 8.7.x.x in MM complaints shot through the roof for me. Aruba helped fine tune some of this for me and upgrading to 8.7.1.3 seems a lot more stable, but I am only working on 2 weeks of it so far. If I had to do over, I would have gone to the latest 8.6 version instead. If I learn anything more today, I will post again. Tim *From:* The EDUCAUSE Wireless Issues Community Group Listserv [mailto: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] *On Behalf Of *Johnson, Christopher *Sent:* Friday, May 21, 2021 10:12 AM *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues Tim, I'm curious what kind of issues you're seeing? We predominately have 225s, 205Hs and 205s - so I'm interested a bit more in what you're seeing. I second the thanks again for all the great feedback/experiences on ArubaOS Versions everyone is sharing. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook and Twitter -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv < WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Rob Harris Sent: Thursday, May 20, 2021 8:49 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv < WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [ mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU ] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 5
RE: [WIRELESS-LAN] Aruba 8.7 issues
Tim, I'm curious what kind of issues you're seeing? We predominately have 225s, 205Hs and 205s - so I'm interested a bit more in what you're seeing. I second the thanks again for all the great feedback/experiences on ArubaOS Versions everyone is sharing. Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook and Twitter -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Rob Harris Sent: Thursday, May 20, 2021 8:49 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7C%7C2d907adbc2584c28faf508d91b960272%7C085f983a0b694270b71d10695076bafe%7C1%7C0%7C637571153364879263%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=W8HPykdxHtmy0S2r89G7DwCda2E7AIaVCMym66VzJTA%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7C%7C2d907adbc2584c28faf508d91b960272%7C085f983a0b694270b71d10695076bafe%7C1%7C0%7C637571153364889258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=Ir9W44SdIGc1%2FvP2wcOTvX0NdK2W6KlUYw%2B%2BFz%2BrzT4%3Dreserved=0 **
RE: [WIRELESS-LAN] Aruba 8.7 issues
The "conservative" branch is considered stable for everyone. We don't go beyond conservative in production unless there's a really compelling reason to. -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 9:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how's the ArubaOS 8.7 code train treating everyone these days? > We're looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) - from > this thread I'm getting a strong "Do Not Engage" vibe. But interested > in everyone's thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7Ce5b7bf5d1d62433f38fd08d91b94654f%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637571146437877600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=8KAYrcvzPrK3vYTrs23SBjCuXxTXDUYRrjSmiT01mBQ%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.educause.edu%2Fcommunitydata=04%7C01%7CRobert.Harris%40CULINARY.EDU%7Ce5b7bf5d1d62433f38fd08d91b94654f%7C91b9485d8b6d4e2da3caf432e56721bd%7C0%7C0%7C637571146437882581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=FAcuaBHmraK%2F3N2fPaOuwJFgG6kfZpPIyoZMLZ1sLPg%3Dreserved=0 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
8.6.0.9 is the latest "conservative" release (what should be their most stable go-to release). Just be sure to check what your environment's min requirements are. For example... for AP-505H, the min software release needed is 8.7.0.0. It's these min req's that usually make you have to go beyond what the latest conservative release is (our case, since we are using 505Hs) Cody -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Tim Tyler Sent: Thursday, May 20, 2021 7:37 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? > We’re looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) – from > this thread I’m getting a strong “Do Not Engage” vibe. But interested > in everyone’s thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
James, Does Aruba state what is the last stable version? I am seriously wondering if going backwards is an option because I am currently seeing some issues as well, but I just upgraded a little over a week ago to 8.7.1.3. We use 325's and 225's predominately. I haven't opened a ticket yet. I was hoping to get through the semester first and then address it. I wonder I others are stable on the latest Aruba version? Tim -Original Message- From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of James Andrewartha Sent: Tuesday, May 18, 2021 11:40 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? > We’re looking at doing some maintenance here shortly and moving from > 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking > at a couple AP-575 APs (which of course requires 8.7 minimum) – from > this thread I’m getting a strong “Do Not Engage” vibe. But interested > in everyone’s thoughts given the additional few months that have > passed since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
Re: [EXT] Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We upgraded to 8.7 when we deployed some 575’s. Had a bunch of client issues and a weird issues with 535’s classifying neighboring ap’s as interfering rouges with 8.7.1.1 and 8.7.1.2. I would recommend skipping 8.7.1.1 and 8.7.1.2.8.7.1.3 has been better. Kevin Grover Utah State University From: The EDUCAUSE Wireless Issues Community Group Listserv on behalf of Iverson, Jeremy Date: Wednesday, May 19, 2021 at 8:28 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [EXT] Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues We too are looking to deploy v8.7.1.3 to support some new Ap503H’s. I wish it was a conservative release, but glad some are not having any issues. Thanks to those who have shared your experiences! Thanks, Jeremy Iverson Northern State University From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Martin Reynolds Sent: Wednesday, May 19, 2021 8:39 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues CAUTION: This email originated from outside of NSU. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username and password. We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion mailto:pertalio...@appstate.edu>> wrote: We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: So how’s the ArubaOS 8.7 code train treating everyone these days? We’re looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking at a couple AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m getting a strong “Do Not Engage” vibe. But interested in everyone’s thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://www.facebook.com/ISUITHelp/> and Twitter<https://twitter.com/ISUITHelp> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all sta
RE: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We too are looking to deploy v8.7.1.3 to support some new Ap503H’s. I wish it was a conservative release, but glad some are not having any issues. Thanks to those who have shared your experiences! Thanks, Jeremy Iverson Northern State University From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Martin Reynolds Sent: Wednesday, May 19, 2021 8:39 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues CAUTION: This email originated from outside of NSU. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username and password. We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion mailto:pertalio...@appstate.edu>> wrote: We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher mailto:cbjo...@ilstu.edu>> wrote: So how’s the ArubaOS 8.7 code train treating everyone these days? We’re looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking at a couple AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m getting a strong “Do Not Engage” vibe. But interested in everyone’s thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://www.facebook.com/ISUITHelp/> and Twitter<https://twitter.com/ISUITHelp> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all stay up. I breathe a bit easier. It appears that in the process of trying to figure out the issues we were seeing, we kept turning up the logging level, which increased the amount of cpu the controller had to spend on logging, to the detriment of processing packets for communication to access points. We still see a few issues with communication between the ap's and the controllers, but now, at least the ap's remain up on the redundant tunnels. Aruba is still working on resolving a load issue on the controller where it's dropping packets. Robert Spellm
Re: [WIRELESS-LAN] [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We are currently running Aruba AP5xx series APs (514, 515, 534, 535) on code level 8.5.0.9 without any issues. We are planning to upgrade to 8.6.0.9 this summer. Thanks, Martin Reynolds Swarthmore College Swarthmore, Pa On Wed, May 19, 2021 at 8:40 AM John Pertalion wrote: > > We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments > planned this summer. > > As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to > 8.7.1.3. We had no issues during this process. > > At this point, we have about 75 AP-503H units deployed and haven't had any > issues with the devices booting up and becoming active on the network. The > rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. > No crashes or lost units since the upgrade. > > > > On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher > wrote: > >> So how’s the ArubaOS 8.7 code train treating everyone these days? We’re >> looking at doing some maintenance here shortly and moving from 8.5.0.11 to >> 8.6 code train for some mini OS enhancements – and looking at a couple >> AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m >> getting a strong “Do Not Engage” vibe. But interested in everyone’s >> thoughts given the additional few months that have passed since then? >> >> *Christopher Johnson* >> Wireless Network Engineer >> Office of Technology Solutions | Illinois State University >> (309) 438-8444 >> >> Stay connected with ISU IT news and tips with @ISU IT Help on Facebook >> <https://www.facebook.com/ISUITHelp/> and Twitter >> <https://twitter.com/ISUITHelp> >> >> *From:* The EDUCAUSE Wireless Issues Community Group Listserv < >> WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Robert Spellman >> *Sent:* Wednesday, December 30, 2020 8:41 PM >> *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU >> *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues >> >> >> >> *[This message came from an external source. If suspicious, report to >> ab...@ilstu.edu ] * >> >> Good news, all of our access points are up tonight. Bad news, it was a >> rough couple of days before we got to this point. A lot of zoom calls with >> Aruba engineers. >> >> >> >> Early on in the debug process, I noticed logs from the access points >> indicating lost packets, duplicate packets, and packets out of sequence. >> Tunnels between the ap's and the controllers weren't stable. Ap's were >> taking hours to boot, or never coming up at all. Ap's that were up and >> passing traffic would drop from the network. >> >> >> >> The first engineer at Aruba said it looks like a network issue, and we >> should look into the network switches between the ap's and the >> controllers. The engineer turned up logging, rebooted controllers and >> access points, and nothing seemed to keep the ap's online for long. I had >> performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the >> controllers back to 8.5.0.3, and this didn't resolve the issue. >> >> >> >> Three more days of tech support. By now, we have upgraded to 8.7.0.0, >> changed logging some more, and collected lots of log files. I install >> devices on both switches which are capturing packets, and we can clearly >> see that all packets from the ap's are arriving properly at the controller, >> which is discarding them. I think we can finally stop blaming the >> network. At this point, we are beyond level one tech support, and >> yesterday, even had developers on the zoom call with us. Then one engineer >> says who turned all this logging on, and turns it all off. Within five >> minutes, all access points are back online. >> >> >> >> We reboot all of the ap's, and within five minutes, they are all back. >> We watch for a few hours, and they all stay up. I breathe a bit easier. >> >> >> >> It appears that in the process of trying to figure out the issues we were >> seeing, we kept turning up the logging level, which increased the amount of >> cpu the controller had to spend on logging, to the detriment of processing >> packets for communication to access points. >> >> >> >> We still see a few issues with communication between the ap's and the >> controllers, but now, at least the ap's remain up on the redundant >> tunnels. Aruba is still working on resolving a load issue on the >> controller where it's dropping packets. >> >> >> >> Robert Spellman >> >> *Associate Director for Network Services* >> >
Re: [External] Re: [WIRELESS-LAN] Aruba 8.7 issues
We upgraded to 8.7.1.3 last Friday. Required because of AP-5xx deployments planned this summer. As advised by Aruba, our upgrade path was 8.3.0.15 to 8.5.0.12 and then to 8.7.1.3. We had no issues during this process. At this point, we have about 75 AP-503H units deployed and haven't had any issues with the devices booting up and becoming active on the network. The rest of our deployment is a mix of 205, 205H, 303H, 315, 27x and 37x APs. No crashes or lost units since the upgrade. On Tue, May 18, 2021 at 5:07 PM Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? We’re > looking at doing some maintenance here shortly and moving from 8.5.0.11 to > 8.6 code train for some mini OS enhancements – and looking at a couple > AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m > getting a strong “Do Not Engage” vibe. But interested in everyone’s > thoughts given the additional few months that have passed since then? > > *Christopher Johnson* > Wireless Network Engineer > Office of Technology Solutions | Illinois State University > (309) 438-8444 > > Stay connected with ISU IT news and tips with @ISU IT Help on Facebook > <https://www.facebook.com/ISUITHelp/> and Twitter > <https://twitter.com/ISUITHelp> > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Robert Spellman > *Sent:* Wednesday, December 30, 2020 8:41 PM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* Re: [WIRELESS-LAN] Aruba 8.7 issues > > > > *[This message came from an external source. If suspicious, report to > ab...@ilstu.edu ] * > > Good news, all of our access points are up tonight. Bad news, it was a > rough couple of days before we got to this point. A lot of zoom calls with > Aruba engineers. > > > > Early on in the debug process, I noticed logs from the access points > indicating lost packets, duplicate packets, and packets out of sequence. > Tunnels between the ap's and the controllers weren't stable. Ap's were > taking hours to boot, or never coming up at all. Ap's that were up and > passing traffic would drop from the network. > > > > The first engineer at Aruba said it looks like a network issue, and we > should look into the network switches between the ap's and the > controllers. The engineer turned up logging, rebooted controllers and > access points, and nothing seemed to keep the ap's online for long. I had > performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the > controllers back to 8.5.0.3, and this didn't resolve the issue. > > > > Three more days of tech support. By now, we have upgraded to 8.7.0.0, > changed logging some more, and collected lots of log files. I install > devices on both switches which are capturing packets, and we can clearly > see that all packets from the ap's are arriving properly at the controller, > which is discarding them. I think we can finally stop blaming the > network. At this point, we are beyond level one tech support, and > yesterday, even had developers on the zoom call with us. Then one engineer > says who turned all this logging on, and turns it all off. Within five > minutes, all access points are back online. > > > > We reboot all of the ap's, and within five minutes, they are all back. We > watch for a few hours, and they all stay up. I breathe a bit easier. > > > > It appears that in the process of trying to figure out the issues we were > seeing, we kept turning up the logging level, which increased the amount of > cpu the controller had to spend on logging, to the detriment of processing > packets for communication to access points. > > > > We still see a few issues with communication between the ap's and the > controllers, but now, at least the ap's remain up on the redundant > tunnels. Aruba is still working on resolving a load issue on the > controller where it's dropping packets. > > > > Robert Spellman > > *Associate Director for Network Services* > > Information and Library Services > > *Bates College* > > p: > > 207-786-6422 > > a: > > 110 Russell Street, Lewiston, ME 04240 > > w: > > www.bates.edu e: rspell...@bates.edu > > > > > > On Tue, Dec 29, 2020 at 8:37 AM Robert Spellman wrote: > > Our latest purchase of Aruba access points included some that required > 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas > break. We have three 7220 controllers and a virtual mobility master > running, with around 1200 access points. > > > > Thursday morning, we did the upgrade on the master and the three > controllers. After a re
Re: [WIRELESS-LAN] Aruba 8.7 issues
On 19/5/21 5:07 am, Johnson, Christopher wrote: > So how’s the ArubaOS 8.7 code train treating everyone these days? We’re > looking at doing some maintenance here shortly and moving from 8.5.0.11 > to 8.6 code train for some mini OS enhancements – and looking at a > couple AP-575 APs (which of course requires 8.7 minimum) – from this > thread I’m getting a strong “Do Not Engage” vibe. But interested in > everyone’s thoughts given the additional few months that have passed > since then? We run mostly 515s (~150) with a ~10 503Hs (which are the reason we went from 8.5.0.11 to 8.7.1.1, now on 8.7.1.3). Since upgrading there's multiple AP crashes per day on both 515 and 503H platforms. There's not a common crash signature, but reading between the lines I think there's some sort of memory leak that is affecting them. TAC have said they have had to go to Broadcom for a fix. Honestly it's not actually too bad since they reboot and come back into service automatically. But I still wouldn't recommend it if you have either model. Also on 8.7.1.1 I had a weird problem with the 515s where they would randomly start getting 50% packet loss, which would clear after a reboot. I haven't seen that since going to 8.7.1.3 40 days ago so I think it's fixed. This one was more of a problem since clients would try to connect and fail and not try another AP, so it actually caused ongoing outages. We also have a 375 and 377 but they've been fine. Thanks, -- James Andrewartha Network & Projects Engineer Christ Church Grammar School Claremont, Western Australia Ph. (08) 9442 1757 Mob. 0424 160 877 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [EXT] Re: [WIRELESS-LAN] Aruba 8.7 issues
We've been running 8.7 for a while. There were a few glitches with the "AP won't boot" issue (seems that in some models, it fails to mark the newly upgraded partition as active, so gets stuck in a boot loop) but the last one was painless. We're on 8.7.1.2 right now, and it's been solid. We'll probably go to the latest 8.7 later this summer, as the version we're on is new enough to have fixes for the array of frag attacks that were recently announced. Frank Sweetser Director of Network Operations Worcester Polytechnic Institute "For every problem, there is a solution that is simple, elegant, and wrong." - HL Mencken From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Johnson, Christopher Sent: Tuesday, May 18, 2021 5:07 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [EXT] Re: [WIRELESS-LAN] Aruba 8.7 issues So how's the ArubaOS 8.7 code train treating everyone these days? We're looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements - and looking at a couple AP-575 APs (which of course requires 8.7 minimum) - from this thread I'm getting a strong "Do Not Engage" vibe. But interested in everyone's thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FISUITHelp%2F=04%7C01%7Cfs%40WPI.EDU%7Ccfdad43549b04ce7c52408d91a40e7f2%7C589c76f5ca1541f9884b55ec15a0672a%7C0%7C0%7C637569688473467072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=FztYJNKjjSxZHXWkncjQE5c1EPvA8EDWO1P%2FJHvh8AM%3D=0> and Twitter<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FISUITHelp=04%7C01%7Cfs%40WPI.EDU%7Ccfdad43549b04ce7c52408d91a40e7f2%7C589c76f5ca1541f9884b55ec15a0672a%7C0%7C0%7C637569688473467072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=qpkKwBZaDPHGSIW%2FAVlZyOXE%2BAFmIJis7m6Y7ETb7Hk%3D=0> From: The EDUCAUSE Wireless Issues Community Group Listserv mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>> On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all stay up. I breathe a bit easier. It appears that in the process of trying to figure out the issues we were seeing, we kept turning up the logging level, which increased the amount of cpu the controller had to spend on logging, to the detriment of processing packets for communication to access points. We still see a few issues with communication between the ap's and the controllers, but now, at least the ap's remain up on the redundant tunnels. Aruba is still working on resolving a load issue on the controller where it's dropping packets. Robert Spellman Associate Director for Network Services Informati
RE: [WIRELESS-LAN] Aruba 8.7 issues
So how’s the ArubaOS 8.7 code train treating everyone these days? We’re looking at doing some maintenance here shortly and moving from 8.5.0.11 to 8.6 code train for some mini OS enhancements – and looking at a couple AP-575 APs (which of course requires 8.7 minimum) – from this thread I’m getting a strong “Do Not Engage” vibe. But interested in everyone’s thoughts given the additional few months that have passed since then? Christopher Johnson Wireless Network Engineer Office of Technology Solutions | Illinois State University (309) 438-8444 Stay connected with ISU IT news and tips with @ISU IT Help on Facebook<https://www.facebook.com/ISUITHelp/> and Twitter<https://twitter.com/ISUITHelp> From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Robert Spellman Sent: Wednesday, December 30, 2020 8:41 PM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues [This message came from an external source. If suspicious, report to ab...@ilstu.edu<mailto:ab...@ilstu.edu>] Good news, all of our access points are up tonight. Bad news, it was a rough couple of days before we got to this point. A lot of zoom calls with Aruba engineers. Early on in the debug process, I noticed logs from the access points indicating lost packets, duplicate packets, and packets out of sequence. Tunnels between the ap's and the controllers weren't stable. Ap's were taking hours to boot, or never coming up at all. Ap's that were up and passing traffic would drop from the network. The first engineer at Aruba said it looks like a network issue, and we should look into the network switches between the ap's and the controllers. The engineer turned up logging, rebooted controllers and access points, and nothing seemed to keep the ap's online for long. I had performed an upgrade from 8.5.0.3 to 8.7.1.1, and we downgraded the controllers back to 8.5.0.3, and this didn't resolve the issue. Three more days of tech support. By now, we have upgraded to 8.7.0.0, changed logging some more, and collected lots of log files. I install devices on both switches which are capturing packets, and we can clearly see that all packets from the ap's are arriving properly at the controller, which is discarding them. I think we can finally stop blaming the network. At this point, we are beyond level one tech support, and yesterday, even had developers on the zoom call with us. Then one engineer says who turned all this logging on, and turns it all off. Within five minutes, all access points are back online. We reboot all of the ap's, and within five minutes, they are all back. We watch for a few hours, and they all stay up. I breathe a bit easier. It appears that in the process of trying to figure out the issues we were seeing, we kept turning up the logging level, which increased the amount of cpu the controller had to spend on logging, to the detriment of processing packets for communication to access points. We still see a few issues with communication between the ap's and the controllers, but now, at least the ap's remain up on the redundant tunnels. Aruba is still working on resolving a load issue on the controller where it's dropping packets. Robert Spellman Associate Director for Network Services Information and Library Services Bates College p: 207-786-6422 a: 110 Russell Street, Lewiston, ME 04240 w: www.bates.edu<http://www.bates.edu/> e: rspell...@bates.edu<mailto:rspellm...@bates.edu> On Tue, Dec 29, 2020 at 8:37 AM Robert Spellman mailto:rsp...@bates.edu>> wrote: Our latest purchase of Aruba access points included some that required 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas break. We have three 7220 controllers and a virtual mobility master running, with around 1200 access points. Thursday morning, we did the upgrade on the master and the three controllers. After a reboot, around 50% of the access points failed to come back online. I figured they would take some time, as they would need to download the new code and reboot, and maybe the controllers were a little busy. By afternoon, we were still around 50%. I opened a ticket with Aruba, and they collected some logs, rebooted a bunch of stuff, but for the most part, didn't make any changes. After rebooting everything, I had around 90% of the access points up. Christmas day, I found access points dropping off of the list. They were still pingable, but the tunnels between the ap's and the controllers wouldn't fully come up. Output from show datapath sessions would show a connection, but not enough to bring the access points back. Saturday, I was back on with Aruba. After another series of log collection and reboots, we were back to around 90%. I still didn't feel good about this, as we hadn't really made any changes. By late evening, I had 9 access points online. Sunday, we moved the aruba co
Re: [WIRELESS-LAN] Aruba 8.7 issues
Rob, When we were troubleshooting an issue last summer, I recall Aruba TAC mentioning problems without out-order-packets. As a troubleshooting step we removed all ECMP and reduced LACP bundles to one member between AP and controller to eliminate this possibility. Just a thought… -j -- Jason Lavoie Director of Networking and Infrastructure Bowdoin College > On Dec 29, 2020, at 11:07 AM, Robert Spellman wrote: > Juniper, huh? We are a Juniper shop too. Odd thing is, we haven't made any > hardware or configuration changes this month. We have been building new > switches for our top of rack, but until Sunday, I hadn't moved the aruba > controllers off of the old layer 2 switches. Once the problem started, I > moved them to the new switches, and the problem continued. > > I do agree that this could be a packet loss issue. Pings from the access > points to the controllers are fine though, even when the access point > connection is down. I've turned on ap-debug on one of the access points in > my office, and I do see a number of messages indicating that the ap received > a packet with the wrong sequence number, as well as messages indicating the > retry timer is getting called. > > In looking at the wrong sequence number errors, it looks like I receive the > response from the controller twice. The ap is fine with the first packet, > but doesn't like the second packet: > > Dec 29 10:26:04 sap_msg_proc: received message from :8222: > code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 > Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id > (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 > Dec 29 10:26:04 PMM: sapd_pwr_event_send_cb: is standby: 0, result OK. > Dec 29 10:26:04 sapd_proc_sap_resp: Calling 'queued' message type 18 Send now > Dec 29 10:26:04 C<< sapd_msg_send_cur: Sending msg ID (,00eb3380,513) > type PWR_EVENT_UPDATE len 182 secure 0 to > Dec 29 10:26:04 sapd_msg_send_cur: Retry Timer called for message > PWR_EVENT_UPDATE, with 10 rexmit time > Dec 29 10:26:04 sapd_svc_recv_sap_msg_output: len: 25 type: PWR_EVENT_UPDATE > seq_num: 512. > Dec 29 10:26:04 sap_msg_proc: received message from :8222: > code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 > Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id > (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 > Dec 29 10:26:04 sapd_proc_sap_resp: Non-matching response received; dropped. > 1 18 != 18, expecting id (,00eb3380,513) -- Jason Lavoie Director of Networking and Infrastructure Bowdoin College (207)725-3315 ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [WIRELESS-LAN] Aruba 8.7 issues
Robert, Our issue was a new juniper QFX VXLAN backbone, so we definitely changed something. We ended up were the ap could ping controller A fine but not controller B, so specific end to end connectivity issues. If you have duplicate packet issues I would start with removing redundancy (disable all redundant links) to make it a single path. That way no possible of duplicate packets. Maybe even collapse down to one controller. Make it super simple and see what happens. From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Robert Spellman Sent: Tuesday, December 29, 2020 11:08 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] Aruba 8.7 issues Hi Jerry. Juniper, huh? We are a Juniper shop too. Odd thing is, we haven't made any hardware or configuration changes this month. We have been building new switches for our top of rack, but until Sunday, I hadn't moved the aruba controllers off of the old layer 2 switches. Once the problem started, I moved them to the new switches, and the problem continued. I do agree that this could be a packet loss issue. Pings from the access points to the controllers are fine though, even when the access point connection is down. I've turned on ap-debug on one of the access points in my office, and I do see a number of messages indicating that the ap received a packet with the wrong sequence number, as well as messages indicating the retry timer is getting called. In looking at the wrong sequence number errors, it looks like I receive the response from the controller twice. The ap is fine with the first packet, but doesn't like the second packet: Dec 29 10:26:04 sap_msg_proc: received message from :8222: code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 Dec 29 10:26:04 PMM: sapd_pwr_event_send_cb: is standby: 0, result OK. Dec 29 10:26:04 sapd_proc_sap_resp: Calling 'queued' message type 18 Send now Dec 29 10:26:04 C<< sapd_msg_send_cur: Sending msg ID (,00eb3380,513) type PWR_EVENT_UPDATE len 182 secure 0 to Dec 29 10:26:04 sapd_msg_send_cur: Retry Timer called for message PWR_EVENT_UPDATE, with 10 rexmit time Dec 29 10:26:04 sapd_svc_recv_sap_msg_output: len: 25 type: PWR_EVENT_UPDATE seq_num: 512. Dec 29 10:26:04 sap_msg_proc: received message from :8222: code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 Dec 29 10:26:04 sapd_proc_sap_resp: Non-matching response received; dropped. 1 18 != 18, expecting id (,00eb3380,513) Robert Spellman Bates College Information and Library Services On Tue, Dec 29, 2020 at 9:44 AM Bucklaew, Jerry mailto:j...@buffalo.edu>> wrote: Robert, We had a similar issue and it turned out to be the Ap’s had communication issues with the controllers and they were constantly re-bootstrapping. The controller showed them down but you could ping them. It was a juniper/backbone thing. But I guess the point is, if the primary or secondary tunnel (to the primary or secondary) controller has packet loss, they will constantly try and re-establish the tunnel and if both go down at the same time, they will rebootstrap. From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>] On Behalf Of Robert Spellman Sent: Tuesday, December 29, 2020 8:38 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU<mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> Subject: [WIRELESS-LAN] Aruba 8.7 issues Our latest purchase of Aruba access points included some that required 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas break. We have three 7220 controllers and a virtual mobility master running, with around 1200 access points. Thursday morning, we did the upgrade on the master and the three controllers. After a reboot, around 50% of the access points failed to come back online. I figured they would take some time, as they would need to download the new code and reboot, and maybe the controllers were a little busy. By afternoon, we were still around 50%. I opened a ticket with Aruba, and they collected some logs, rebooted a bunch of stuff, but for the most part, didn't make any changes. After rebooting everything, I had around 90% of the access points up. Christmas day, I found access points dropping off of the list. They were still pingable, but the tunnels between the ap's and the controllers wouldn't fully come up. Output from show datapath sessions would show a connection, but not enough to bring the access points back. Saturday, I was back on with Aruba. After another series of log collection and reboots, we were back to a
Re: [WIRELESS-LAN] Aruba 8.7 issues
Hi Jerry. Juniper, huh? We are a Juniper shop too. Odd thing is, we haven't made any hardware or configuration changes this month. We have been building new switches for our top of rack, but until Sunday, I hadn't moved the aruba controllers off of the old layer 2 switches. Once the problem started, I moved them to the new switches, and the problem continued. I do agree that this could be a packet loss issue. Pings from the access points to the controllers are fine though, even when the access point connection is down. I've turned on ap-debug on one of the access points in my office, and I do see a number of messages indicating that the ap received a packet with the wrong sequence number, as well as messages indicating the retry timer is getting called. In looking at the wrong sequence number errors, it looks like I receive the response from the controller twice. The ap is fine with the first packet, but doesn't like the second packet: Dec 29 10:26:04 sap_msg_proc: received message from :8222: code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 Dec 29 10:26:04 PMM: sapd_pwr_event_send_cb: is standby: 0, result OK. Dec 29 10:26:04 sapd_proc_sap_resp: Calling 'queued' message type 18 Send now Dec 29 10:26:04 C<< sapd_msg_send_cur: Sending msg ID (,00eb3380,513) type PWR_EVENT_UPDATE len 182 secure 0 to Dec 29 10:26:04 sapd_msg_send_cur: Retry Timer called for message PWR_EVENT_UPDATE, with 10 rexmit time Dec 29 10:26:04 sapd_svc_recv_sap_msg_output: len: 25 type: PWR_EVENT_UPDATE seq_num: 512. Dec 29 10:26:04 sap_msg_proc: received message from :8222: code 16101 type PWR_EVENT_UPDATE id (,00eb3380,512) length 5 Dec 29 10:26:04 C>> sapd_proc_sap_resp: msg from :8222 id (,00eb3380,512) type PWR_EVENT_UPDATE result 0 size 0 Dec 29 10:26:04 sapd_proc_sap_resp: Non-matching response received; dropped. 1 18 != 18, expecting id (,00eb3380,513) Robert Spellman Bates College Information and Library Services On Tue, Dec 29, 2020 at 9:44 AM Bucklaew, Jerry wrote: > Robert, > > > > We had a similar issue and it turned out to be the Ap’s had > communication issues with the controllers and they were constantly > re-bootstrapping. The controller showed them down but you could ping > them. It was a juniper/backbone thing. But I guess the point is, if the > primary or secondary tunnel (to the primary or secondary) controller has > packet loss, they will constantly try and re-establish the tunnel and if > both go down at the same time, they will rebootstrap. > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv [mailto: > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] *On Behalf Of *Robert Spellman > *Sent:* Tuesday, December 29, 2020 8:38 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* [WIRELESS-LAN] Aruba 8.7 issues > > > > Our latest purchase of Aruba access points included some that required > 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas > break. We have three 7220 controllers and a virtual mobility master > running, with around 1200 access points. > > > > Thursday morning, we did the upgrade on the master and the three > controllers. After a reboot, around 50% of the access points failed to > come back online. I figured they would take some time, as they would need > to download the new code and reboot, and maybe the controllers were a > little busy. > > > > By afternoon, we were still around 50%. I opened a ticket with Aruba, and > they collected some logs, rebooted a bunch of stuff, but for the most part, > didn't make any changes. After rebooting everything, I had around 90% of > the access points up. > > > > Christmas day, I found access points dropping off of the list. They were > still pingable, but the tunnels between the ap's and the controllers > wouldn't fully come up. Output from show datapath sessions would show a > connection, but not enough to bring the access points back. > > > > Saturday, I was back on with Aruba. After another series of log > collection and reboots, we were back to around 90%. I still didn't feel > good about this, as we hadn't really made any changes. By late evening, I > had 9 access points online. > > > > Sunday, we moved the aruba controllers to new switches. They had > previously been connected to a stack of switches, all on the same layer 2 > network. After the reboot, we again had most of the access points online, > but as the evening wore on, the number of access points online dropped. > > > > Monday, call Aruba, wash, rinse and repeat. At one point Monday > afternoon, we had all access points back online. I don't think the > engineer made any c
Re: [WIRELESS-LAN] [EXTERNAL] [WIRELESS-LAN] Aruba 8.7 issues
aruba-master is set in dns, and resolves to the vrrp ip address shared by the three controllers. As for setting the LMS-IP, I looked in the system-profile default, which is what we are using, and no LMS-IP is set. I don't think we use provisioning profiles. We did look at the license usage, and the number of consumed licenses is pretty close to the number of active ap's, and we are currently licensed for 1536, so we have plenty of license room. No access points show up when running show ap database fags L. On Tue, Dec 29, 2020 at 9:17 AM Turpin, Max wrote: > Robert, > > > > I’ve seen similar behavior but it happened during a cluster rebuild prior > to a code upgrade. Unfortunately, it’s been a while and I was not able to > find my notes from that case. What I do remember is that like your issue, > the APs were pingable, but they would not join the cluster. I ended up > having to shut/no shut the interfaces in order for them to rejoin MM. > > > > As for your setup, how do your APs find your cluster? Do you have > aruba-master resolve to the cluster VIP? Are you setting the LMS-IP on the > ap system profile or ap group? Are you using option 43 in DHCP? Do you use > provisioning profiles in activate? > > > > Silly question as well, but what does your license table look like? I have > seen licensing issues where licenses get used by a controller that is in an > error state so the MM sees the pool as exhausted. You can look for L flags > in the ap database. > > > > Just a few things to consider. > > > > Max Turpin > > > > *From:* The EDUCAUSE Wireless Issues Community Group Listserv < > WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> *On Behalf Of *Robert Spellman > *Sent:* Tuesday, December 29, 2020 8:38 AM > *To:* WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU > *Subject:* [EXTERNAL] [WIRELESS-LAN] Aruba 8.7 issues > > > > Our latest purchase of Aruba access points included some that required > 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas > break. We have three 7220 controllers and a virtual mobility master > running, with around 1200 access points. > > > > Thursday morning, we did the upgrade on the master and the three > controllers. After a reboot, around 50% of the access points failed to > come back online. I figured they would take some time, as they would need > to download the new code and reboot, and maybe the controllers were a > little busy. > > > > By afternoon, we were still around 50%. I opened a ticket with Aruba, and > they collected some logs, rebooted a bunch of stuff, but for the most part, > didn't make any changes. After rebooting everything, I had around 90% of > the access points up. > > > > Christmas day, I found access points dropping off of the list. They were > still pingable, but the tunnels between the ap's and the controllers > wouldn't fully come up. Output from show datapath sessions would show a > connection, but not enough to bring the access points back. > > > > Saturday, I was back on with Aruba. After another series of log > collection and reboots, we were back to around 90%. I still didn't feel > good about this, as we hadn't really made any changes. By late evening, I > had 9 access points online. > > > > Sunday, we moved the aruba controllers to new switches. They had > previously been connected to a stack of switches, all on the same layer 2 > network. After the reboot, we again had most of the access points online, > but as the evening wore on, the number of access points online dropped. > > > > Monday, call Aruba, wash, rinse and repeat. At one point Monday > afternoon, we had all access points back online. I don't think the > engineer made any configuration changes, just rebooted the controllers. > > > > Right now, 622 are down, 607 are up. > > > > We did revert to 8.5.0.3, but that didn't seem to help. Right now, we are > at 8.7.0.0. > > > > > > Robert Spellman > > Bates College > > Information and Library Services > > ** > Replies to EDUCAUSE Community Group emails are sent to the entire > community list. If you want to reply only to the person who sent the > message, copy and paste their email address and forward the email reply. > Additional participation and subscription information can be found at > https://www.educause.edu/community > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.educause.edu_community=DwMFaQ=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U=zobI7d8a-PnWsDxhdheA-Pkovo0vk-DVRBlpbuIQ8mE=oqX6gtCO9lCl7XrYhLDWJ8JKmV3beRRw76Ae4m0cMho=8IvpxA2eHKw7AF7GCok-Lr3TpMOHc0espuWJRB3X67U=> > > ** > Replies to EDUCAUSE Community Group emails are sen
RE: [WIRELESS-LAN] Aruba 8.7 issues
Robert, We had a similar issue and it turned out to be the Ap’s had communication issues with the controllers and they were constantly re-bootstrapping. The controller showed them down but you could ping them. It was a juniper/backbone thing. But I guess the point is, if the primary or secondary tunnel (to the primary or secondary) controller has packet loss, they will constantly try and re-establish the tunnel and if both go down at the same time, they will rebootstrap. From: The EDUCAUSE Wireless Issues Community Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Robert Spellman Sent: Tuesday, December 29, 2020 8:38 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [WIRELESS-LAN] Aruba 8.7 issues Our latest purchase of Aruba access points included some that required 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas break. We have three 7220 controllers and a virtual mobility master running, with around 1200 access points. Thursday morning, we did the upgrade on the master and the three controllers. After a reboot, around 50% of the access points failed to come back online. I figured they would take some time, as they would need to download the new code and reboot, and maybe the controllers were a little busy. By afternoon, we were still around 50%. I opened a ticket with Aruba, and they collected some logs, rebooted a bunch of stuff, but for the most part, didn't make any changes. After rebooting everything, I had around 90% of the access points up. Christmas day, I found access points dropping off of the list. They were still pingable, but the tunnels between the ap's and the controllers wouldn't fully come up. Output from show datapath sessions would show a connection, but not enough to bring the access points back. Saturday, I was back on with Aruba. After another series of log collection and reboots, we were back to around 90%. I still didn't feel good about this, as we hadn't really made any changes. By late evening, I had 9 access points online. Sunday, we moved the aruba controllers to new switches. They had previously been connected to a stack of switches, all on the same layer 2 network. After the reboot, we again had most of the access points online, but as the evening wore on, the number of access points online dropped. Monday, call Aruba, wash, rinse and repeat. At one point Monday afternoon, we had all access points back online. I don't think the engineer made any configuration changes, just rebooted the controllers. Right now, 622 are down, 607 are up. We did revert to 8.5.0.3, but that didn't seem to help. Right now, we are at 8.7.0.0. Robert Spellman Bates College Information and Library Services ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
RE: [EXTERNAL] [WIRELESS-LAN] Aruba 8.7 issues
Robert, I’ve seen similar behavior but it happened during a cluster rebuild prior to a code upgrade. Unfortunately, it’s been a while and I was not able to find my notes from that case. What I do remember is that like your issue, the APs were pingable, but they would not join the cluster. I ended up having to shut/no shut the interfaces in order for them to rejoin MM. As for your setup, how do your APs find your cluster? Do you have aruba-master resolve to the cluster VIP? Are you setting the LMS-IP on the ap system profile or ap group? Are you using option 43 in DHCP? Do you use provisioning profiles in activate? Silly question as well, but what does your license table look like? I have seen licensing issues where licenses get used by a controller that is in an error state so the MM sees the pool as exhausted. You can look for L flags in the ap database. Just a few things to consider. Max Turpin From: The EDUCAUSE Wireless Issues Community Group Listserv On Behalf Of Robert Spellman Sent: Tuesday, December 29, 2020 8:38 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: [EXTERNAL] [WIRELESS-LAN] Aruba 8.7 issues Our latest purchase of Aruba access points included some that required 8.7, so we planned on upgrading from 8.5.0.3 to 8.7.1.0 over Christmas break. We have three 7220 controllers and a virtual mobility master running, with around 1200 access points. Thursday morning, we did the upgrade on the master and the three controllers. After a reboot, around 50% of the access points failed to come back online. I figured they would take some time, as they would need to download the new code and reboot, and maybe the controllers were a little busy. By afternoon, we were still around 50%. I opened a ticket with Aruba, and they collected some logs, rebooted a bunch of stuff, but for the most part, didn't make any changes. After rebooting everything, I had around 90% of the access points up. Christmas day, I found access points dropping off of the list. They were still pingable, but the tunnels between the ap's and the controllers wouldn't fully come up. Output from show datapath sessions would show a connection, but not enough to bring the access points back. Saturday, I was back on with Aruba. After another series of log collection and reboots, we were back to around 90%. I still didn't feel good about this, as we hadn't really made any changes. By late evening, I had 9 access points online. Sunday, we moved the aruba controllers to new switches. They had previously been connected to a stack of switches, all on the same layer 2 network. After the reboot, we again had most of the access points online, but as the evening wore on, the number of access points online dropped. Monday, call Aruba, wash, rinse and repeat. At one point Monday afternoon, we had all access points back online. I don't think the engineer made any configuration changes, just rebooted the controllers. Right now, 622 are down, 607 are up. We did revert to 8.5.0.3, but that didn't seem to help. Right now, we are at 8.7.0.0. Robert Spellman Bates College Information and Library Services ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.educause.edu_community=DwMFaQ=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U=zobI7d8a-PnWsDxhdheA-Pkovo0vk-DVRBlpbuIQ8mE=oqX6gtCO9lCl7XrYhLDWJ8JKmV3beRRw76Ae4m0cMho=8IvpxA2eHKw7AF7GCok-Lr3TpMOHc0espuWJRB3X67U=> ** Replies to EDUCAUSE Community Group emails are sent to the entire community list. If you want to reply only to the person who sent the message, copy and paste their email address and forward the email reply. Additional participation and subscription information can be found at https://www.educause.edu/community
