Re: why does network manager start activating disconnected ethernet ports?
On Sat, 2018-11-10 at 13:54 -0500, David P. Reed wrote: > Dan - thanks! Yes, the ports were in LOWER_UP, now that I understand > what that means. > > It is not a Network Manager problem. It was a real screwup on my > part. Sorry to waste your time. No problem, glad it got figured out! Dan > For some reason two of the ports were actually cabled to a switch > that was otherwise unused (not connecting anywhere). It was supposed > to be powered off, but it turns out that someone (not me, that I know > of) turned it on. That brought the links up, but there was no DHCP > server reachable. > > Again, thanks for the help with my troubleshooting. > > I do like NM a lot, and appreciate your effort put in to continue > improving and supporting it. > > David > -Original Message- > From: "Dan Williams" > Sent: Thursday, November 8, 2018 5:35pm > To: "David P. Reed" > Cc: networkmanager-list@gnome.org > Subject: Re: why does network manager start activating disconnected > ethernet ports? > > > > On Thu, 2018-11-08 at 16:16 -0500, David P. Reed wrote: > > Hi Dan - > > I tried the rpm -e command and it said I don't have that module > > installed. I did grep the directories (/var/run/NetworkManager and > > /etc/NetworkManager) recursively for the string "carrier" and found > > nothing there. So that seems not to be the problem. > > > > It's clear from the journalctl log entries that the two non- > > connected > > interfaces are being "auto-activated" over and over by something. > > They move from prepare->config->ip-config. > > Reason is "none", but I think that's normally what goes on. > > > > One personal theory (that I don't know how to test properly) is > > that > > the RealTek r8169 driver (which seems to have been demonstrating > > problems on Ubuntu in the past few months, on some hardware) is > > somehow saying the ports are coming online or are online when they > > are not. I'm not sure how NetworkManager gets told about such > > transitions, but it might be a problem. I also wonder about chrony > > somehow trying to probe the interfaces, triggering them by > > accident. > > NetworkManager listens to kernel messages from the driver. Run "ip > link show dev " which should look similar to this: > > 2: enp0s31f6: mtu 1500 > qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 > link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff > > if you see LOWER_UP that means the kernel driver thinks there is a > carrier/link. At that point it's a kernel driver bug if nothing is > plugged into the card's port. > > If it doesn't show LOWER_UP, then it's probably some configuration > issue or bug in NM itself. > > Dan > > > Is there a "no-ignore-carrier" option that would be something to > > try > > for debugging? or some specific logging option that will tell more > > about why these ports are "auto-activating"? > > > > I have several other Fedora 28 machines running similar > > configurations, but they don't seem to do this. They are different, > > in that their Ethernet hardware is different, but the software > > setup > > is roughly similar. > > > > Anyway, I'd love to figure this out. I can share the journals if > > you > > want or get other data, but this level of logging doesn't say very > > much to me other than what I summarize above. > > > > Thanks. > > -Original Message- > > From: "Dan Williams" > > Sent: Wednesday, November 7, 2018 5:55pm > > To: "David P. Reed" , networkmanager-list@gnom > > e. > > org > > Subject: Re: why does network manager start activating disconnected > > ethernet ports? > > > > > > > > On Wed, 2018-11-07 at 16:23 -0500, David P. Reed wrote: > > > I have a Fedora 28 server in my lab that has multiple network > > > adapters, but only one of them is cabled to a switch. All the > > > software is updated to NetworkManager-1.10.12-1.fc28. > > > > Did you install the Fedora 28 "Server" spin, or something like > > Workstation? Some of the config options are different for different > > variants. That includes installing the "00-server.conf" file into > > /etc/NetworkManager/conf.d for Server installs in some cases. > > > > One thing that does is set "ignore-carrier=*" which, as you may > > guess, > > makes NM ignore the carrier detection of the NIC on all interfaces. > > Mostly for servers with sta
Re: why does network manager start activating disconnected ethernet ports?
Dan - thanks! Yes, the ports were in LOWER_UP, now that I understand what that means. It is not a Network Manager problem. It was a real screwup on my part. Sorry to waste your time. For some reason two of the ports were actually cabled to a switch that was otherwise unused (not connecting anywhere). It was supposed to be powered off, but it turns out that someone (not me, that I know of) turned it on. That brought the links up, but there was no DHCP server reachable. Again, thanks for the help with my troubleshooting. I do like NM a lot, and appreciate your effort put in to continue improving and supporting it. David -Original Message- From: "Dan Williams" Sent: Thursday, November 8, 2018 5:35pm To: "David P. Reed" Cc: networkmanager-list@gnome.org Subject: Re: why does network manager start activating disconnected ethernet ports? On Thu, 2018-11-08 at 16:16 -0500, David P. Reed wrote: > Hi Dan - > I tried the rpm -e command and it said I don't have that module > installed. I did grep the directories (/var/run/NetworkManager and > /etc/NetworkManager) recursively for the string "carrier" and found > nothing there. So that seems not to be the problem. > > It's clear from the journalctl log entries that the two non-connected > interfaces are being "auto-activated" over and over by something. > They move from prepare->config->ip-config. > Reason is "none", but I think that's normally what goes on. > > One personal theory (that I don't know how to test properly) is that > the RealTek r8169 driver (which seems to have been demonstrating > problems on Ubuntu in the past few months, on some hardware) is > somehow saying the ports are coming online or are online when they > are not. I'm not sure how NetworkManager gets told about such > transitions, but it might be a problem. I also wonder about chrony > somehow trying to probe the interfaces, triggering them by accident. NetworkManager listens to kernel messages from the driver. Run "ip link show dev " which should look similar to this: 2: enp0s31f6: mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff if you see LOWER_UP that means the kernel driver thinks there is a carrier/link. At that point it's a kernel driver bug if nothing is plugged into the card's port. If it doesn't show LOWER_UP, then it's probably some configuration issue or bug in NM itself. Dan > Is there a "no-ignore-carrier" option that would be something to try > for debugging? or some specific logging option that will tell more > about why these ports are "auto-activating"? > > I have several other Fedora 28 machines running similar > configurations, but they don't seem to do this. They are different, > in that their Ethernet hardware is different, but the software setup > is roughly similar. > > Anyway, I'd love to figure this out. I can share the journals if you > want or get other data, but this level of logging doesn't say very > much to me other than what I summarize above. > > Thanks. > -Original Message----- > From: "Dan Williams" > Sent: Wednesday, November 7, 2018 5:55pm > To: "David P. Reed" , networkmanager-list@gnome. > org > Subject: Re: why does network manager start activating disconnected > ethernet ports? > > > > On Wed, 2018-11-07 at 16:23 -0500, David P. Reed wrote: > > I have a Fedora 28 server in my lab that has multiple network > > adapters, but only one of them is cabled to a switch. All the > > software is updated to NetworkManager-1.10.12-1.fc28. > > Did you install the Fedora 28 "Server" spin, or something like > Workstation? Some of the config options are different for different > variants. That includes installing the "00-server.conf" file into > /etc/NetworkManager/conf.d for Server installs in some cases. > > One thing that does is set "ignore-carrier=*" which, as you may > guess, > makes NM ignore the carrier detection of the NIC on all interfaces. > Mostly for servers with static IPs where connectivity shouldn't > change > even if you unplug or trip over the cable. > > Try 'rpm -e NetworkManager-config-server' and then restarting NM or > rebooting and see if that fixes it. > > Dan > > > However, when I run journalctl -e I typically see pages and pages > > and > > pages of DHCPDISCOVER requests on the non-connected ports, > > constantly > > retrying. The logs fill very, very fast. > > > > I did something yesterday that managed to cause the retrying to > > stop, > > leaving the unused ports all in the disconnected state (nmcli > > device
Re: why does network manager start activating disconnected ethernet ports?
On Thu, 2018-11-08 at 16:16 -0500, David P. Reed wrote: > Hi Dan - > I tried the rpm -e command and it said I don't have that module > installed. I did grep the directories (/var/run/NetworkManager and > /etc/NetworkManager) recursively for the string "carrier" and found > nothing there. So that seems not to be the problem. > > It's clear from the journalctl log entries that the two non-connected > interfaces are being "auto-activated" over and over by something. > They move from prepare->config->ip-config. > Reason is "none", but I think that's normally what goes on. > > One personal theory (that I don't know how to test properly) is that > the RealTek r8169 driver (which seems to have been demonstrating > problems on Ubuntu in the past few months, on some hardware) is > somehow saying the ports are coming online or are online when they > are not. I'm not sure how NetworkManager gets told about such > transitions, but it might be a problem. I also wonder about chrony > somehow trying to probe the interfaces, triggering them by accident. NetworkManager listens to kernel messages from the driver. Run "ip link show dev " which should look similar to this: 2: enp0s31f6: mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff if you see LOWER_UP that means the kernel driver thinks there is a carrier/link. At that point it's a kernel driver bug if nothing is plugged into the card's port. If it doesn't show LOWER_UP, then it's probably some configuration issue or bug in NM itself. Dan > Is there a "no-ignore-carrier" option that would be something to try > for debugging? or some specific logging option that will tell more > about why these ports are "auto-activating"? > > I have several other Fedora 28 machines running similar > configurations, but they don't seem to do this. They are different, > in that their Ethernet hardware is different, but the software setup > is roughly similar. > > Anyway, I'd love to figure this out. I can share the journals if you > want or get other data, but this level of logging doesn't say very > much to me other than what I summarize above. > > Thanks. > -----Original Message- > From: "Dan Williams" > Sent: Wednesday, November 7, 2018 5:55pm > To: "David P. Reed" , networkmanager-list@gnome. > org > Subject: Re: why does network manager start activating disconnected > ethernet ports? > > > > On Wed, 2018-11-07 at 16:23 -0500, David P. Reed wrote: > > I have a Fedora 28 server in my lab that has multiple network > > adapters, but only one of them is cabled to a switch. All the > > software is updated to NetworkManager-1.10.12-1.fc28. > > Did you install the Fedora 28 "Server" spin, or something like > Workstation? Some of the config options are different for different > variants. That includes installing the "00-server.conf" file into > /etc/NetworkManager/conf.d for Server installs in some cases. > > One thing that does is set "ignore-carrier=*" which, as you may > guess, > makes NM ignore the carrier detection of the NIC on all interfaces. > Mostly for servers with static IPs where connectivity shouldn't > change > even if you unplug or trip over the cable. > > Try 'rpm -e NetworkManager-config-server' and then restarting NM or > rebooting and see if that fixes it. > > Dan > > > However, when I run journalctl -e I typically see pages and pages > > and > > pages of DHCPDISCOVER requests on the non-connected ports, > > constantly > > retrying. The logs fill very, very fast. > > > > I did something yesterday that managed to cause the retrying to > > stop, > > leaving the unused ports all in the disconnected state (nmcli > > device > > status said disconnected for each one). I thought, hmm... well that > > problem is now fixed. `journalctl -e` showed everything quiet. > > > > Today, I thought I'd check. When I did `journalctl -e` there were > > no > > DHCPDISCOVERs issued, and when I did `nmcli device status` the > > ports > > were all still "disconnected". > > > > But then I did `nmcli con show`. It said the ports were in a > > disconnected state. > > > > However at this point I happened to repeat the `journalctl -e` > > command, and my goodness - the stream of DHCPDISCOVER requests > > timing > > out were a sight to amaze, and the network manager kept > > transitioning > > the state of the ports that had no cables correspondingly, trying > > to > > get an IP address. > >
Re: why does network manager start activating disconnected ethernet ports?
Hi Dan - I tried the rpm -e command and it said I don't have that module installed. I did grep the directories (/var/run/NetworkManager and /etc/NetworkManager) recursively for the string "carrier" and found nothing there. So that seems not to be the problem. It's clear from the journalctl log entries that the two non-connected interfaces are being "auto-activated" over and over by something. They move from prepare->config->ip-config. Reason is "none", but I think that's normally what goes on. One personal theory (that I don't know how to test properly) is that the RealTek r8169 driver (which seems to have been demonstrating problems on Ubuntu in the past few months, on some hardware) is somehow saying the ports are coming online or are online when they are not. I'm not sure how NetworkManager gets told about such transitions, but it might be a problem. I also wonder about chrony somehow trying to probe the interfaces, triggering them by accident. Is there a "no-ignore-carrier" option that would be something to try for debugging? or some specific logging option that will tell more about why these ports are "auto-activating"? I have several other Fedora 28 machines running similar configurations, but they don't seem to do this. They are different, in that their Ethernet hardware is different, but the software setup is roughly similar. Anyway, I'd love to figure this out. I can share the journals if you want or get other data, but this level of logging doesn't say very much to me other than what I summarize above. Thanks. -Original Message- From: "Dan Williams" Sent: Wednesday, November 7, 2018 5:55pm To: "David P. Reed" , networkmanager-list@gnome.org Subject: Re: why does network manager start activating disconnected ethernet ports? On Wed, 2018-11-07 at 16:23 -0500, David P. Reed wrote: > I have a Fedora 28 server in my lab that has multiple network > adapters, but only one of them is cabled to a switch. All the > software is updated to NetworkManager-1.10.12-1.fc28. Did you install the Fedora 28 "Server" spin, or something like Workstation? Some of the config options are different for different variants. That includes installing the "00-server.conf" file into /etc/NetworkManager/conf.d for Server installs in some cases. One thing that does is set "ignore-carrier=*" which, as you may guess, makes NM ignore the carrier detection of the NIC on all interfaces. Mostly for servers with static IPs where connectivity shouldn't change even if you unplug or trip over the cable. Try 'rpm -e NetworkManager-config-server' and then restarting NM or rebooting and see if that fixes it. Dan > However, when I run journalctl -e I typically see pages and pages and > pages of DHCPDISCOVER requests on the non-connected ports, constantly > retrying. The logs fill very, very fast. > > I did something yesterday that managed to cause the retrying to stop, > leaving the unused ports all in the disconnected state (nmcli device > status said disconnected for each one). I thought, hmm... well that > problem is now fixed. `journalctl -e` showed everything quiet. > > Today, I thought I'd check. When I did `journalctl -e` there were no > DHCPDISCOVERs issued, and when I did `nmcli device status` the ports > were all still "disconnected". > > But then I did `nmcli con show`. It said the ports were in a > disconnected state. > > However at this point I happened to repeat the `journalctl -e` > command, and my goodness - the stream of DHCPDISCOVER requests timing > out were a sight to amaze, and the network manager kept transitioning > the state of the ports that had no cables correspondingly, trying to > get an IP address. > > This may be my misunderstanding, but the interface knows whether the > port is connected to some other port with a cable or if it is not. It > seems logical also that merely asking for the connection status > shouldn't "turn on" a lot of useless DHCP discovery on disconnected > ports. > > So am I confused or is this a bug? > > Tell me what you need to find it if so. > > I'd also like to know how to stop the behavior. I know I could change > the ports to not be "auto", but I'd really like to be able to just > plug a cable in and start using the port to talk to some other system > or switch. > ___ > networkmanager-list mailing list > networkmanager-list@gnome.org > https://mail.gnome.org/mailman/listinfo/networkmanager-list___ networkmanager-list mailing list networkmanager-list@gnome.org https://mail.gnome.org/mailman/listinfo/networkmanager-list
