Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
> Thank you for your assistance. With hint for the relevant man > page "bridge-utils-interfaces" I found the bridge setup working > using the configuration > > auto br0 > iface br0 inet static > address 192.168.1.1 > network 192.168.1.0 > netmask 255.255.255.0 > bridge_ports none > bridge_hw 86:aa:aa:aa:aa:aa > > With that there is no race observed, I deem this bug report as > invalid. Yes, that looks ok for a setup where you don't want to add any ports to a bridge. > I tested without the legacy stuff, worked, making this bug report > irrelevant. Testing how far the change can be backported is > done on demand later, not relevant here (Bullseye). I don't remember any reason why current bullseye package can't be used at buster or prior versions. Regards. -- Manty/BestiaTester -> http://manty.net
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
Hello Santiago Garcia, Dennis, Thank you for your assistance. With hint for the relevant man page "bridge-utils-interfaces" I found the bridge setup working using the configuration auto br0 iface br0 inet static address 192.168.1.1 network 192.168.1.0 netmask 255.255.255.0 bridge_ports none bridge_hw 86:aa:aa:aa:aa:aa With that there is no race observed, I deem this bug report as invalid. Understanding now with your help the interactions of udev/systemd, I will split the automation script that worked for years to one variant for old setups/non-systemd machines and use the new features for new machines. Santiago Garcia Mantinan writes: > Hi! > > First I'd like to thank Dennis for his good support, as always > ;-) > >> > $ ifup virtbr0 > Cannot find device "virtbr0" > ifup: failed >> to bring up virtbr0 >> >> It is because the "bridge_ports" directive is missing. From >> the manpage bridge-utils-interfaces(5): >> >> bridge_ports interface specification this option must exist >> for the scripts to setup the bridge. >> >> Either specify "bridge_ports none" or "bridge_ports enp2s0 >> enp2s1" (or whatever your physical interfaces are named). > > That's it, you always have to specify the bridge_ports directive > so that we treat the interface as a bridge. > >> > I also reactivated "systemd-udevd": > ... >> > # systemctl reload /lib/systemd/network/80-bridgeutils.link >> > Failed to reload >> lib-systemd-network-80\x2dbridgeutils.link.mount: Unit >> lib-systemd-network-80\x2dbridgeutils.link.mount not found. > > I really believe that this contribution from Dennis should > not be needed, so I'd appreciate if you could test without > this extra stuff, which hasn't really been thoroughtly tested > and test with the standard setup, if we identify a problem > with the standard bridge_hw setup we'll go over it. I tested without the legacy stuff, worked, making this bug report irrelevant. Testing how far the change can be backported is done on demand later, not relevant here (Bullseye). > If you test it like that, please provide feedback to know if > it worked and if we can close the bug or not. It worked, please close the report. I would have closed it myself according to https://www.debian.org/Bugs/Developer#closing if I would have been sure if "invalid" reports are closed the same way as "done" ones. Kind regards, Roman
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
Hi! First I'd like to thank Dennis for his good support, as always ;-) > > $ ifup virtbr0 > > Cannot find device "virtbr0" > > ifup: failed to bring up virtbr0 > > It is because the "bridge_ports" directive is missing. From the > manpage bridge-utils-interfaces(5): > >bridge_ports interface specification > this option must exist for the scripts to setup the bridge. > > Either specify "bridge_ports none" or "bridge_ports enp2s0 enp2s1" (or > whatever your physical interfaces are named). That's it, you always have to specify the bridge_ports directive so that we treat the interface as a bridge. > > I also reactivated "systemd-udevd": ... > > # systemctl reload /lib/systemd/network/80-bridgeutils.link > > Failed to reload lib-systemd-network-80\x2dbridgeutils.link.mount: Unit > > lib-systemd-network-80\x2dbridgeutils.link.mount not found. I really believe that this contribution from Dennis should not be needed, so I'd appreciate if you could test without this extra stuff, which hasn't really been thoroughtly tested and test with the standard setup, if we identify a problem with the standard bridge_hw setup we'll go over it. If you test it like that, please provide feedback to know if it worked and if we can close the bug or not. Regards. -- Manty/BestiaTester -> http://manty.net
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
X-Debbugs-CC: Roman Fiedler On Sat, Aug 14, 2021 at 08:18:00PM +, Roman Fiedler wrote: > > iface virtbr0 inet static > > address 192.168.1.1 > > netmask 255.255.255.0 > > bridge_hw 86:aa:aa:aa:aa:aa > > Weird, using the configuration from above will result in: > > $ ifup virtbr0 > Cannot find device "virtbr0" > ifup: failed to bring up virtbr0 It is because the "bridge_ports" directive is missing. From the manpage bridge-utils-interfaces(5): bridge_ports interface specification this option must exist for the scripts to setup the bridge. Either specify "bridge_ports none" or "bridge_ports enp2s0 enp2s1" (or whatever your physical interfaces are named). > I also reactivated "systemd-udevd": > > $ systemctl start systemd-udevd-kernel.socket > $ systemctl start systemd-udevd-control.socket > $ systemctl start systemd-udevd.service > > And then tried to use the link, but I am not sure, if it is active > without rebooting as "reload" does not seem to be the right command > for it. > > # systemctl reload /lib/systemd/network/80-bridgeutils.link > Failed to reload lib-systemd-network-80\x2dbridgeutils.link.mount: Unit > lib-systemd-network-80\x2dbridgeutils.link.mount not found. Running "systemctl daemon-reload" would have been the correct thing to do here. > Without rebooting (which at the moment would be annoying for > the test machine), the "hw_address" does still not work: > > iface virtbr0 inet static > address 192.168.1.1 > netmask 255.255.255.0 > pre-up brctl addbr virtbr0 > post-down brctl delbr virtbr0 > bridge_hw 86:aa:aa:aa:aa:aa > > but now the MAC seems truely random each time the bridge is created: This is because with systemd-udevd stopped (or told to not touch bridge devices through 80-bridgeutils.link) the MAC address the kernel assigns upon device creation (which is randomly generated every time) remains untouched. systemd-udevd would also assign a random MAC address, but generates the same one every time (it's random only compared to the burnt-in MAC address). Regards.
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
Hello,
Thanks for your swift reply and really helpful information!
Dennis Filder writes:
> X-Debbugs-CC: Roman Fiedler , Michael Tokarev
>
>
> As stated in the documentation, bridge-utils recently added support
> for the "bridge_hw" directive which allows you define a MAC address
> for a bridge interface. Changing your stanza to just this should
> already work:
>
> iface virtbr0 inet static
> address 192.168.1.1
> netmask 255.255.255.0
> bridge_hw 86:aa:aa:aa:aa:aa
Weird, using the configuration from above will result in:
$ ifup virtbr0
Cannot find device "virtbr0"
ifup: failed to bring up virtbr0
Modifying above stanza to this ...
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
pre-up brctl addbr virtbr0
post-down brctl delbr virtbr0
bridge_hw 86:aa:aa:aa:aa:aa
... will make "ifup virtbr0" succeed, but the MAC is not set
to the expected value:
$ ifup virtbr0
$ ip link show virtbr0
9: virtbr0: mtu 1500 qdisc noqueue state
DOWN mode DEFAULT group default qlen 1000
link/ether 42:dc:6a:57:58:bd brd ff:ff:ff:ff:ff:ff
> If you also use systemd, then systemd-udevd may cause trouble because
> in its default configuration it will assign a randomly generated MAC
> address to the bridge device which might cause a race.
I don't know if relevant to this case but the MAC assigned
(42:dc:6a:57:58:bd) is not random, it is the same one every time
the bridge is created.
> I haven't tried this, but putting this udev rule into
> /etc/udev/rules.d/95-bridge.rules should prevent systemd-udevd from
> touching any bridge interfaces at all:
>
>ACTION=="add", SUBSYSTEM=="net", DEVTYPE=="bridge", ENV{TAGS}-="systemd"
As "systemd-udevd" is active ...
# ps aux | grep udev
root 207 0.0 0.9 21892 4808 ?Ss 08:33 0:00
/lib/systemd/systemd-udevd
... I just stopped it completely using ...
$ systemctl stop systemd-udevd-kernel.socket
$ systemctl stop systemd-udevd-control.socket
$ systemctl stop systemd-udevd.service
... to avoid any interference. With
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
pre-up brctl addbr virtbr0
post-down brctl delbr virtbr0
bridge_hw 86:aa:aa:aa:aa:aa
this will still not work even with udev disabled:
$ ifup virtbr0; ip link show virtbr0
13: virtbr0: mtu 1500 qdisc noqueue state
UNKNOWN mode DEFAULT group default qlen 1000
link/ether 8a:ad:e6:c7:ab:76 brd ff:ff:ff:ff:ff:ff
But without "systemd-udevd" my initial configuration will work
also in the virtual machine:
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
pre-up brctl addbr virtbr0
pre-up ip link set virtbr0 address 86:aa:aa:aa:aa:aa
pre-up ip link set virtbr0 up
post-down ip link set virtbr0 down
post-down brctl delbr virtbr0
$ ifup virtbr0; ip link show virtbr0
12: virtbr0: mtu 1500 qdisc noqueue state
DOWN mode DEFAULT group default qlen 1000
link/ether 86:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
> Alternatively placing the file 80-bridge-utils.link from #991416
> (message #17)[0] into /lib/systemd/network/ should work as well.
I also reactivated "systemd-udevd":
$ systemctl start systemd-udevd-kernel.socket
$ systemctl start systemd-udevd-control.socket
$ systemctl start systemd-udevd.service
And then tried to use the link, but I am not sure, if it is active
without rebooting as "reload" does not seem to be the right command
for it.
# systemctl reload /lib/systemd/network/80-bridgeutils.link
Failed to reload lib-systemd-network-80\x2dbridgeutils.link.mount: Unit
lib-systemd-network-80\x2dbridgeutils.link.mount not found.
Without rebooting (which at the moment would be annoying for
the test machine), the "hw_address" does still not work:
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
pre-up brctl addbr virtbr0
post-down brctl delbr virtbr0
bridge_hw 86:aa:aa:aa:aa:aa
but now the MAC seems truely random each time the bridge is created:
$ ifup virtbr0; ip link show virtbr0
17: virtbr0: mtu 1500 qdisc noqueue state
UNKNOWN mode DEFAULT group default qlen 1000
link/ether 3e:6b:bc:8a:b0:68 brd ff:ff:ff:ff:ff:ff
...
link/ether 82:da:00:1c:98:ab brd ff:ff:ff:ff:ff:ff
...
Instead the config
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
pre-up brctl addbr virtbr0
pre-up ip link set virtbr0 address 86:aa:aa:aa:aa:aa
pre-up ip link set virtbr0 up
post-down ip link set virtbr0 down
post-down brctl delbr virtbr0
now works without any specific udev rules and systemd services running
normally.
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
X-Debbugs-CC: Roman Fiedler , Michael Tokarev
As stated in the documentation, bridge-utils recently added support
for the "bridge_hw" directive which allows you define a MAC address
for a bridge interface. Changing your stanza to just this should
already work:
iface virtbr0 inet static
address 192.168.1.1
netmask 255.255.255.0
bridge_hw 86:aa:aa:aa:aa:aa
If you also use systemd, then systemd-udevd may cause trouble because
in its default configuration it will assign a randomly generated MAC
address to the bridge device which might cause a race.
I haven't tried this, but putting this udev rule into
/etc/udev/rules.d/95-bridge.rules should prevent systemd-udevd from
touching any bridge interfaces at all:
ACTION=="add", SUBSYSTEM=="net", DEVTYPE=="bridge", ENV{TAGS}-="systemd"
Alternatively placing the file 80-bridge-utils.link from #991416
(message #17)[0] into /lib/systemd/network/ should work as well.
Regards,
Dennis Filder
0: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991416#17
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
On Sat, 14 Aug 2021 08:28:41 + Roman Fiedler wrote: Package: bridge-utils Version: 1.7-1 Severity: serious When running "brctl addbr" and "ip link set [if] address" immediately afterwards, the second command will fail to apply the address change. This is somehow annoying as the MAC would be used in security related filtering and monitoring of the network traffic, which then fails. The configuration from "/etc/network/interfaces" reliably triggering the bug is: auto virtbr0 iface virtbr0 inet static address 192.168.1.1 netmask 255.255.255.0 pre-up brctl addbr virtbr0 pre-up ip link set virtbr0 address 86:aa:aa:aa:aa:aa pre-up ip link set virtbr0 up post-down ip link set virtbr0 down post-down brctl delbr virtbr0 How about switching from brctl to ip, once you already use it anyway? So instead of brctl addbr virtbr0, it will read ip link add virtbr0 type bridge, and ip link del virbr0, something like that. I don't know about brctl, - we stopped using this utility some 10 years ago due to numerous issues. Also I don't know whenever it's a good idea to mark this bug as serious. Thanks, /mjt
Bug#992158: Race in ifup maybe related to brctl failure in pre-up of network interface
Package: bridge-utils Version: 1.7-1 Severity: serious When running "brctl addbr" and "ip link set [if] address" immediately afterwards, the second command will fail to apply the address change. This is somehow annoying as the MAC would be used in security related filtering and monitoring of the network traffic, which then fails. The configuration from "/etc/network/interfaces" reliably triggering the bug is: auto virtbr0 iface virtbr0 inet static address 192.168.1.1 netmask 255.255.255.0 pre-up brctl addbr virtbr0 pre-up ip link set virtbr0 address 86:aa:aa:aa:aa:aa pre-up ip link set virtbr0 up post-down ip link set virtbr0 down post-down brctl delbr virtbr0 Running "ifdown virtbr0; ifup virtbr0; ip link show" will report link/ether 86:8a:6a:ee:5e:a2 brd ff:ff:ff:ff:ff:ff so the setting to "86:aa:aa:aa:aa:aa" does not take effect. The reason I expect the race to be in brctl itself or related a race in the kernel just exposed by brctl is, that following changes will produce correct results: * "strace -o dump ifup virtbr0" eliminates the behavior. * Using " pre-up brctl addbr virtbr0 && sleep 1" will make "ifup" wait for one second, correct MAC is set. Instead when changing the initial interfaces configuration to include pre-up ip link set virtbr0 address 86:aa:aa:aa:aa:aa || touch /root/fail "ifup" will still expose the bug but /root/fail is not created. So if "ip link" fails to react correctly on any intermediate state when the bridge is coming up, then at least it does not detect it correctly and report it via an error code. As attaching a debugger eliminates the bug I have no idea how to quickly rule out such an effect. Weirdly this bug does not occur on a multicore real-hardware machine (I have no single core hardware or tried to run Linux single core via boot options) but till now only in single core qemu machines (I did not test multi-core qemu yet). But the core count, kernel behaviour might be just a red herring. Any ideas how that could happen? Maybe would "brctl add" need to wait for a confirmation from kernel, that the bridge setup is completed before exiting?
