Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
Are all of the access points on the same channels? >• Problem seems to be localized in (four classrooms) Bob On Sep 1, 2020, at 7:42 PM, Debbie Unterseher <0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: We have had poor wifi at our university since school started August 10. We have less students than we did last semester, and less students in classes because of social distancing. I am not the network person. However I know I have had good luck at finding answers from other people, so I thought I would share this with you all to see if you have any input. Most of this means nothing to me. Would be happy for any suggestions you have! Below are the two emails that the IT department just sent - one to me and one to the whole campus. Thanks again for any input. ~~ Let me just say in non-technical terms, I have heard that there is a point that the traffic through the access points just stops, and my understanding is that the Ubiquiti APs get super hot and some have failed. Some will work after cooling down for several minutes. The HP and Ubiquitis are reacting the same, but the Ubiquitis are worse. We did just switch from Moodle to Canvas, and some classes are being taught via Zoom. ~~ Information Systems would like to give an update on what we have found, so far, in our work to solve the WiFi problems that have been experienced this semester. Our working theory is that over the summer updates to the networking for computers and mobile devices have changed to include new features. Occasionally, the new features cause problems with our local infrastructure. Right now our wireless network is overloaded with a lot of unnecessary traffic. On a small network with a few devices such as a home, this traffic would not be a problem and is very useful for interacting with printers, cameras, or other devices. On a large network with 1800 devices just on the student network, it can be a big problem. We are trying to resolve how to control this traffic. This does not have anything to do with our Internet connection speed, which is doing very well. It is mainly centered around activity happening on the student network, which of course is our largest network. We have been continually testing and making changes to our network. Some of you may have seen us monitoring classes or even asking people in a class to turn off or on certain devices. While there is not much activity for us to monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are busy times for testing. We made some significant configuration changes and will be testing them tomorrow. Below are some details for the technically curious: Here is what we know: * Very high volume of multicast traffic * Seems to be mostly mDNS protocol * IPv4 and IPv6 are used for transport * Affects WiFi more than wired network * Affects both HP and Ubiquiti devices Here is what we think: * Clients have been updated to use newer protocols * mDNS is used mostly to talk to IoT devices * mDNS is similar to AppleTalk's NBP and is very chatty * Our Ubiquiti APs fail, some may not be useful for production anymore * HP APs that are 802.11ac compatible fail but will recover * It seems that 802.11n units are more resilient or at least they can recover on their own * Problem seems to be localized in (four classrooms) Here is what we are doing: * Upgraded firmware on our switches * Changed WiFiTX protection to "No MAC protection" which excludes 803.11b devices * Turned on Spanning Tree and IGMP helpers to WiFi * Changed DTIM to 3 * Downgraded the firmware on our working Ubiquiti APs * Experimenting with replacing all 802.11ac units with .11n devices * Controlling broadcasts at switch and AP level * Disabled mDNS at switch level for IPv4 on capable switches * Trying to disable mDNS over IPv6 at switch level * Considering requesting all clients stop using IPv6 Have received permission from Nicole to disrupt Nursing classes in LLC. Will do so if action is relevant We have done some experiments with turning off all devices or just phones, but need more data. Email directly to me from networking guy: Here is an article that explains the type of thing we think may be causing problems on our WiFi: https://framebyframewifi.net/2018/01/15/beware-of-mdns-floods-from-buggy-android-clients/ I have done a number of packet captures and found in problem areas we have a high percentage of mdns traffic (Multicast DNS, basically peer to peer chatting and device discovery. As best we can tell we do not need any mdns on our campus for academic functions. Some IoT things in the dorms on the Student network many need this, but not much else. Please note that this is not a problem just for Android on our networks however, in fact most of the
Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
Are you sure that "No MAC protection” has disabled the lower data rates. Multicast is sent at the lowest data rate. So things that reduce multicast traffic or raise the rate of multicast traffic should help. https://help.ui.com/hc/en-us/articles/115006559827-UniFi-802-11-Basic-Supported-Rate-Controls > On Sep 1, 2020, at 7:32 PM, Debbie Unterseher > <0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: > > We have had poor wifi at our university since school started August 10. We > have less students than we did last semester, and less students in classes > because of social distancing. I am not the network person. However I know I > have had good luck at finding answers from other people, so I thought I would > share this with you all to see if you have any input. Most of this means > nothing to me. Would be happy for any suggestions you have! Below are the two > emails that the IT department just sent - one to me and one to the whole > campus. Thanks again for any input. > ~~ > Let me just say in non-technical terms, I have heard that there is a point > that the traffic through the access points just stops, and my understanding > is that the Ubiquiti APs get super hot and some have failed. Some will work > after cooling down for several minutes. The HP and Ubiquitis are reacting the > same, but the Ubiquitis are worse. We did just switch from Moodle to Canvas, > and some classes are being taught via Zoom. > > ~~ > Information Systems would like to give an update on what we have found, so > far, in our work to solve the WiFi problems that have been experienced this > semester. > > Our working theory is that over the summer updates to the networking for > computers and mobile devices have changed to include new features. > Occasionally, the new features cause problems with our local infrastructure. > Right now our wireless network is overloaded with a lot of unnecessary > traffic. On a small network with a few devices such as a home, this traffic > would not be a problem and is very useful for interacting with printers, > cameras, or other devices. On a large network with 1800 devices just on the > student network, it can be a big problem. We are trying to resolve how to > control this traffic. This does not have anything to do with our Internet > connection speed, which is doing very well. It is mainly centered around > activity happening on the student network, which of course is our largest > network. > > We have been continually testing and making changes to our network. Some of > you may have seen us monitoring classes or even asking people in a class to > turn off or on certain devices. While there is not much activity for us to > monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are busy > times for testing. We made some significant configuration changes and will be > testing them tomorrow. > > Below are some details for the technically curious: > > Here is what we know: > • Very high volume of multicast traffic > • Seems to be mostly mDNS protocol > • IPv4 and IPv6 are used for transport > • Affects WiFi more than wired network > • Affects both HP and Ubiquiti devices > > Here is what we think: > • Clients have been updated to use newer protocols > • mDNS is used mostly to talk to IoT devices > • mDNS is similar to AppleTalk's NBP and is very chatty > • Our Ubiquiti APs fail, some may not be useful for production anymore > • HP APs that are 802.11ac compatible fail but will recover > • It seems that 802.11n units are more resilient or at least they can > recover on their own > • Problem seems to be localized in (four classrooms) > > Here is what we are doing: > • Upgraded firmware on our switches > • Changed WiFiTX protection to "No MAC protection" which excludes > 803.11b devices > • Turned on Spanning Tree and IGMP helpers to WiFi > • Changed DTIM to 3 > • Downgraded the firmware on our working Ubiquiti APs > • Experimenting with replacing all 802.11ac units with .11n devices > • Controlling broadcasts at switch and AP level > • Disabled mDNS at switch level for IPv4 on capable switches > • Trying to disable mDNS over IPv6 at switch level > • Considering requesting all clients stop using IPv6 > Have received permission from Nicole to disrupt Nursing classes in LLC. Will > do so if action is relevant > We have done some experiments with turning off all devices or just phones, > but need more data. > > > Email directly to me from networking guy: > > Here is an article that explains the type of thing we think may be > causing problems on our WiFi: > https://framebyframewifi.net/2018/01/15/beware-of-mdns-floods-from-buggy-android-clients/ > > I have done a number of packet
Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
We have had the “convert multicast to unicast or broadcast” over our wireless, disabled, for quite some time. We leave that functionality to the LAN. __ __ Fishel Erps, Sr. Network & Infrastructure Engineer School of Visual Arts 136 W 21st St., 8th Floor New York, NY, 10011 LL: 212-592-2416 E: fe...@sva.edu ___ Please excuse any typographical errors as this e-mail has been sent from my mobile device ___ On Sep 1, 2020, at 21:00, Lee H Badman < 00db5b77bd95-dmarc-requ...@listserv.educause.edu> wrote: Has anyone simply tried to disable multicast on the WLAN? Lee Badman (mobile) On Sep 1, 2020, at 8:42 PM, Debbie Unterseher < 0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: We have had poor wifi at our university since school started August 10. We have less students than we did last semester, and less students in classes because of social distancing. I am not the network person. However I know I have had good luck at finding answers from other people, so I thought I would share this with you all to see if you have any input. Most of this means nothing to me. Would be happy for any suggestions you have! Below are the two emails that the IT department just sent - one to me and one to the whole campus. Thanks again for any input. ~~ Let me just say in non-technical terms, I have heard that there is a point that the traffic through the access points just stops, and my understanding is that the Ubiquiti APs get super hot and some have failed. Some will work after cooling down for several minutes. The HP and Ubiquitis are reacting the same, but the Ubiquitis are worse. We did just switch from Moodle to Canvas, and some classes are being taught via Zoom. ~~ Information Systems would like to give an update on what we have found, so far, in our work to solve the WiFi problems that have been experienced this semester. Our working theory is that over the summer updates to the networking for computers and mobile devices have changed to include new features. Occasionally, the new features cause problems with our local infrastructure. Right now our wireless network is overloaded with a lot of unnecessary traffic. On a small network with a few devices such as a home, this traffic would not be a problem and is very useful for interacting with printers, cameras, or other devices. On a large network with 1800 devices just on the student network, it can be a big problem. We are trying to resolve how to control this traffic. This does not have anything to do with our Internet connection speed, which is doing very well. It is mainly centered around activity happening on the student network, which of course is our largest network. We have been continually testing and making changes to our network. Some of you may have seen us monitoring classes or even asking people in a class to turn off or on certain devices. While there is not much activity for us to monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are busy times for testing. We made some significant configuration changes and will be testing them tomorrow. Below are some details for the technically curious: Here is what we know: - Very high volume of multicast traffic - Seems to be mostly mDNS protocol - IPv4 and IPv6 are used for transport - Affects WiFi more than wired network - Affects both HP and Ubiquiti devices Here is what we think: - Clients have been updated to use newer protocols - mDNS is used mostly to talk to IoT devices - mDNS is similar to AppleTalk's NBP and is very chatty - Our Ubiquiti APs fail, some may not be useful for production anymore - HP APs that are 802.11ac compatible fail but will recover - It seems that 802.11n units are more resilient or at least they can recover on their own - Problem seems to be localized in (four classrooms) Here is what we are doing: - Upgraded firmware on our switches - Changed WiFiTX protection to "No MAC protection" which excludes 803.11b devices - Turned on Spanning Tree and IGMP helpers to WiFi - Changed DTIM to 3 - Downgraded the firmware on our working Ubiquiti APs - Experimenting with replacing all 802.11ac units with .11n devices - Controlling broadcasts at switch and AP level - Disabled mDNS at switch level for IPv4 on capable switches - Trying to disable mDNS over IPv6 at switch level - Considering requesting all clients stop using IPv6 Have received permission from Nicole to disrupt Nursing classes in LLC. Will do so if action is relevant We have done some experiments with turning off all devices or just phones, but need more data. Email directly to me from networking guy: Here is an article that explains the type of thing we think may be causing
Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
This is exactly what came to mind, Ricardo: Debbie, it sounds like you have too large a broadcast area: it needs to be split up into smaller segments. This is usually done with a combination of two things: first, the DHCP server needs to be set up to accommodate non-overlapping IP pools for the new VLANs you're going to create, then the network switches need to be reconfigured with the additional vlans and set up to forward DHCP traffic to your DHCP server. (Sorry if this is pedantic, but it sounds like you might want to know some of this detail.) I am surprised to hear the Ubiquiti are failing, because our tests showed them to be stable. But each WAP does have to process any mDNS broadcast it 'hears' in its VLAN, and they may be simply using all their CPU to process these broadcasts. John Rodkey Director of Servers and Networks Westmont College Verification: Unsure if this is a legitimate email to an email list? Make sure it is recorded at https://my.westmont.edu/it_emails "*God-fearing faith... is neither brash nor foolhardy and does not tempt God."* - Martin Luther On Tue, Sep 1, 2020 at 5:53 PM Ricardo Stella wrote: > > I remember when the iPad came out and it was the hottest ticket item > during the winter break. Spring started, and WiFi went down the drain on a > Monday. > > At the time, we had a flat network with all wireless devices in the same > vlan. By Wednesday we had isolated all wifi by dorm, creating a vlan for > each one of them, and separate from the wired network. > > At the time we had Bluesocket APs. That helped tremendously, but within > two years, we revamped the whole wireless network with Aruba, and they have > "AirGroups" which basically create isolation between users for mDNS. > > > On Tue, Sep 1, 2020 at 8:42 PM Debbie Unterseher < > 0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: > >> We have had poor wifi at our university since school started August 10. >> We have less students than we did last semester, and less students in >> classes because of social distancing. I am not the network person. However >> I know I have had good luck at finding answers from other people, so I >> thought I would share this with you all to see if you have any input. Most >> of this means nothing to me. Would be happy for any suggestions you have! >> Below are the two emails that the IT department just sent - one to me and >> one to the whole campus. Thanks again for any input. >> ~~ >> Let me just say in non-technical terms, I have heard that there is a >> point that the traffic through the access points just stops, and my >> understanding is that the Ubiquiti APs get super hot and some have failed. >> Some will work after cooling down for several minutes. The HP and Ubiquitis >> are reacting the same, but the Ubiquitis are worse. We did just switch >> from Moodle to Canvas, and some classes are being taught via Zoom. >> >> ~~ >> Information Systems would like to give an update on what we have found, >> so far, in our work to solve the WiFi problems that have been experienced >> this semester. >> >> Our working theory is that over the summer updates to the networking for >> computers and mobile devices have changed to include new features. >> Occasionally, the new features cause problems with our local >> infrastructure. Right now our wireless network is overloaded with a lot of >> unnecessary traffic. On a small network with a few devices such as a home, >> this traffic would not be a problem and is very useful for interacting with >> printers, cameras, or other devices. On a large network with 1800 devices >> just on the student network, it can be a big problem. We are trying to >> resolve how to control this traffic. This does not have anything to do with >> our Internet connection speed, which is doing very well. It is mainly >> centered around activity happening on the student network, which of course >> is our largest network. >> >> We have been continually testing and making changes to our network. Some >> of you may have seen us monitoring classes or even asking people in a class >> to turn off or on certain devices. While there is not much activity for us >> to monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are >> busy times for testing. We made some significant configuration changes and >> will be testing them tomorrow. >> >> Below are some details for the technically curious: >> >> Here is what we know: >> >>- Very high volume of multicast traffic >>- Seems to be mostly mDNS protocol >>- IPv4 and IPv6 are used for transport >>- Affects WiFi more than wired network >>- Affects both HP and Ubiquiti devices >> >> >> Here is what we think: >> >>- Clients have been updated to use newer protocols >>- mDNS is used mostly to talk to IoT devices >>- mDNS is similar to AppleTalk's NBP and is very chatty >>- Our Ubiquiti APs fail, some may not be useful for production
Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
Has anyone simply tried to disable multicast on the WLAN? Lee Badman (mobile) On Sep 1, 2020, at 8:42 PM, Debbie Unterseher <0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: We have had poor wifi at our university since school started August 10. We have less students than we did last semester, and less students in classes because of social distancing. I am not the network person. However I know I have had good luck at finding answers from other people, so I thought I would share this with you all to see if you have any input. Most of this means nothing to me. Would be happy for any suggestions you have! Below are the two emails that the IT department just sent - one to me and one to the whole campus. Thanks again for any input. ~~ Let me just say in non-technical terms, I have heard that there is a point that the traffic through the access points just stops, and my understanding is that the Ubiquiti APs get super hot and some have failed. Some will work after cooling down for several minutes. The HP and Ubiquitis are reacting the same, but the Ubiquitis are worse. We did just switch from Moodle to Canvas, and some classes are being taught via Zoom. ~~ Information Systems would like to give an update on what we have found, so far, in our work to solve the WiFi problems that have been experienced this semester. Our working theory is that over the summer updates to the networking for computers and mobile devices have changed to include new features. Occasionally, the new features cause problems with our local infrastructure. Right now our wireless network is overloaded with a lot of unnecessary traffic. On a small network with a few devices such as a home, this traffic would not be a problem and is very useful for interacting with printers, cameras, or other devices. On a large network with 1800 devices just on the student network, it can be a big problem. We are trying to resolve how to control this traffic. This does not have anything to do with our Internet connection speed, which is doing very well. It is mainly centered around activity happening on the student network, which of course is our largest network. We have been continually testing and making changes to our network. Some of you may have seen us monitoring classes or even asking people in a class to turn off or on certain devices. While there is not much activity for us to monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are busy times for testing. We made some significant configuration changes and will be testing them tomorrow. Below are some details for the technically curious: Here is what we know: * Very high volume of multicast traffic * Seems to be mostly mDNS protocol * IPv4 and IPv6 are used for transport * Affects WiFi more than wired network * Affects both HP and Ubiquiti devices Here is what we think: * Clients have been updated to use newer protocols * mDNS is used mostly to talk to IoT devices * mDNS is similar to AppleTalk's NBP and is very chatty * Our Ubiquiti APs fail, some may not be useful for production anymore * HP APs that are 802.11ac compatible fail but will recover * It seems that 802.11n units are more resilient or at least they can recover on their own * Problem seems to be localized in (four classrooms) Here is what we are doing: * Upgraded firmware on our switches * Changed WiFiTX protection to "No MAC protection" which excludes 803.11b devices * Turned on Spanning Tree and IGMP helpers to WiFi * Changed DTIM to 3 * Downgraded the firmware on our working Ubiquiti APs * Experimenting with replacing all 802.11ac units with .11n devices * Controlling broadcasts at switch and AP level * Disabled mDNS at switch level for IPv4 on capable switches * Trying to disable mDNS over IPv6 at switch level * Considering requesting all clients stop using IPv6 Have received permission from Nicole to disrupt Nursing classes in LLC. Will do so if action is relevant We have done some experiments with turning off all devices or just phones, but need more data. Email directly to me from networking guy: Here is an article that explains the type of thing we think may be causing problems on our WiFi: https://framebyframewifi.net/2018/01/15/beware-of-mdns-floods-from-buggy-android-clients/ I have done a number of packet captures and found in problem areas we have a high percentage of mdns traffic (Multicast DNS, basically peer to peer chatting and device discovery. As best we can tell we do not need any mdns on our campus for academic functions. Some IoT things in the dorms on the Student network many need this, but not much else. Please note that this is not a problem just for Android on our networks however, in fact most of the mdns traffic we are seeing is more
Re: [WIRELESS-LAN] MDNS Traffic - problem with wifi on campus
I remember when the iPad came out and it was the hottest ticket item during the winter break. Spring started, and WiFi went down the drain on a Monday. At the time, we had a flat network with all wireless devices in the same vlan. By Wednesday we had isolated all wifi by dorm, creating a vlan for each one of them, and separate from the wired network. At the time we had Bluesocket APs. That helped tremendously, but within two years, we revamped the whole wireless network with Aruba, and they have "AirGroups" which basically create isolation between users for mDNS. On Tue, Sep 1, 2020 at 8:42 PM Debbie Unterseher < 0058e3b52c23-dmarc-requ...@listserv.educause.edu> wrote: > We have had poor wifi at our university since school started August 10. We > have less students than we did last semester, and less students in classes > because of social distancing. I am not the network person. However I know I > have had good luck at finding answers from other people, so I thought I > would share this with you all to see if you have any input. Most of this > means nothing to me. Would be happy for any suggestions you have! Below are > the two emails that the IT department just sent - one to me and one to the > whole campus. Thanks again for any input. > ~~ > Let me just say in non-technical terms, I have heard that there is a point > that the traffic through the access points just stops, and my understanding > is that the Ubiquiti APs get super hot and some have failed. Some will work > after cooling down for several minutes. The HP and Ubiquitis are reacting > the same, but the Ubiquitis are worse. We did just switch from Moodle to > Canvas, and some classes are being taught via Zoom. > > ~~ > Information Systems would like to give an update on what we have found, so > far, in our work to solve the WiFi problems that have been experienced this > semester. > > Our working theory is that over the summer updates to the networking for > computers and mobile devices have changed to include new features. > Occasionally, the new features cause problems with our local > infrastructure. Right now our wireless network is overloaded with a lot of > unnecessary traffic. On a small network with a few devices such as a home, > this traffic would not be a problem and is very useful for interacting with > printers, cameras, or other devices. On a large network with 1800 devices > just on the student network, it can be a big problem. We are trying to > resolve how to control this traffic. This does not have anything to do with > our Internet connection speed, which is doing very well. It is mainly > centered around activity happening on the student network, which of course > is our largest network. > > We have been continually testing and making changes to our network. Some > of you may have seen us monitoring classes or even asking people in a class > to turn off or on certain devices. While there is not much activity for us > to monitor on Tuesdays and Thursdays, Mondays, Wednesdays, and Fridays are > busy times for testing. We made some significant configuration changes and > will be testing them tomorrow. > > Below are some details for the technically curious: > > Here is what we know: > >- Very high volume of multicast traffic >- Seems to be mostly mDNS protocol >- IPv4 and IPv6 are used for transport >- Affects WiFi more than wired network >- Affects both HP and Ubiquiti devices > > > Here is what we think: > >- Clients have been updated to use newer protocols >- mDNS is used mostly to talk to IoT devices >- mDNS is similar to AppleTalk's NBP and is very chatty >- Our Ubiquiti APs fail, some may not be useful for production anymore >- HP APs that are 802.11ac compatible fail but will recover >- It seems that 802.11n units are more resilient or at least they can >recover on their own >- Problem seems to be localized in (four classrooms) > > > Here is what we are doing: > >- Upgraded firmware on our switches >- Changed WiFiTX protection to "No MAC protection" which excludes >803.11b devices >- Turned on Spanning Tree and IGMP helpers to WiFi >- Changed DTIM to 3 >- Downgraded the firmware on our working Ubiquiti APs >- Experimenting with replacing all 802.11ac units with .11n devices >- Controlling broadcasts at switch and AP level >- Disabled mDNS at switch level for IPv4 on capable switches >- Trying to disable mDNS over IPv6 at switch level >- Considering requesting all clients stop using IPv6 > > Have received permission from Nicole to disrupt Nursing classes in LLC. > Will do so if action is relevant > We have done some experiments with turning off all devices or just phones, > but need more data. > > > Email directly to me from networking guy: > > Here is an article that explains the type of thing we think may be
