I can confirm this behavior after update to systemd 237-3ubuntu10.39 on privileged and unprivileged containers. We classify this bug as critical because in the next monthly update cycle via ansible orchestration we would expect failing all Bionic LXC containers. The only workaround seems to manually define a static network configuration in /etc/netplan/10-lxc.yaml via lxc-attach. But this is not an acceptable solution for 180 servers we run.
Some our Plesk servers already failed completely due to automatic upgrades Plesk triggered daily. Websites were down for hours until we figured out the reason. I would appreciate that someone solves this problem due to urgency. Thank you in advance. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1863873 Title: Systemd fails to configure bridged network in LXC container Status in systemd package in Ubuntu: Incomplete Bug description: In all our LXC containers running Bionic Beaver, installing systemd 237-3ubuntu10.39 results in losing network configuration. It is still possible to configure the network "by hand" with /usr/sbin/ip, but of course, the configuration is lost at reboot. An example is provided, followed by a complete procedure to reproduce the issue. Affected container distributions ================================ Xenial Xerus systemd 229-4ubuntu21.27: OK, not affected Bionic Beaver systemd 237-3ubuntu10.38: OK, not affected Bionic Beaver systemd 237-3ubuntu10.39: BUGGY Disco Dingo systemd 240-6ubuntu5.8: OK, not affected Eoan Ermine systemd 242-7ubuntu3.6: OK, not affected Eoan Ermine systemd 242-7ubuntu3.7: BUGGY Focal Fossa systemd 244.2-1ubuntu1: BUGGY Affected hosts ============== Debian Buster with default 4.19.0-6-amd64, custom 5.3.9, 5.4.8 or 5.4.13 kernel Ubuntu 16.04 lxc 2.0.8-0ubuntu1~16.04.2 (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1863873/comments/7) Example ======= Example host bridge configuration --------------------------------- 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000 link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br1 state DOWN group default qlen 1000 link/ether 00:25:90:2b:f1:61 brd ff:ff:ff:ff:ff:ff 4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:25:90:2b:f1:60 brd ff:ff:ff:ff:ff:ff inet 192.168.252.24/24 brd 192.168.252.255 scope global br0 valid_lft forever preferred_lft forever inet 192.168.193.203/24 brd 192.168.193.255 scope global br0:1 valid_lft forever preferred_lft forever inet6 fe80::225:90ff:fe2b:f160/64 scope link valid_lft forever preferred_lft forever Example container network configuration --------------------------------------- lxc.net.0.type = veth lxc.net.0.veth.pair = vps525389 lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.hwaddr = 02:00:00:52:53:89 lxc.net.0.name = eth0 lxc.net.0.ipv4.gateway = 192.168.252.1 lxc.net.0.ipv4.address = 192.168.252.177/32 Example steps to reproduce, inside the container ------------------------------------------------ root@vps525389:~# lsb_release -rd Description: Ubuntu 18.04.4 LTS Release: 18.04 root@vps525389:~# apt-cache policy systemd systemd: Installed: 237-3ubuntu10.38 Candidate: 237-3ubuntu10.39 Version table: 237-3ubuntu10.39 500 500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages *** 237-3ubuntu10.38 500 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 100 /var/lib/dpkg/status 237-3ubuntu10 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages root@vps525389:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1958: eth0@if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.252.177/32 brd 255.255.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 xxxx:xxxx:x:xx::x:xxxx/128 scope global valid_lft forever preferred_lft forever inet6 xxxx::xx:xxxx:xxxx/64 scope link valid_lft forever preferred_lft forever root@vps525389:~# apt install systemd Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libnss-systemd libpam-systemd libsystemd0 Suggested packages: systemd-container policykit-1 The following packages will be upgraded: libnss-systemd libpam-systemd libsystemd0 systemd 4 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. Need to get 3330 kB of archives. After this operation, 7168 B of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnss-systemd amd64 237-3ubuntu10.39 [104 kB] Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpam-systemd amd64 237-3ubuntu10.39 [107 kB] Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.39 [2912 kB] Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd0 amd64 237-3ubuntu10.39 [206 kB] Fetched 3330 kB in 3s (1274 kB/s) (Reading database ... 18195 files and directories currently installed.) Preparing to unpack .../libnss-systemd_237-3ubuntu10.39_amd64.deb ... Unpacking libnss-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ... Preparing to unpack .../libpam-systemd_237-3ubuntu10.39_amd64.deb ... Unpacking libpam-systemd:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ... Preparing to unpack .../systemd_237-3ubuntu10.39_amd64.deb ... Unpacking systemd (237-3ubuntu10.39) over (237-3ubuntu10.38) ... Preparing to unpack .../libsystemd0_237-3ubuntu10.39_amd64.deb ... Unpacking libsystemd0:amd64 (237-3ubuntu10.39) over (237-3ubuntu10.38) ... Setting up libsystemd0:amd64 (237-3ubuntu10.39) ... Setting up systemd (237-3ubuntu10.39) ... Setting up libnss-systemd:amd64 (237-3ubuntu10.39) ... Setting up libpam-systemd:amd64 (237-3ubuntu10.39) ... Processing triggers for dbus (1.12.2-1ubuntu1.1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... root@vps525389:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 1958: eth0@if1959: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 02:00:00:52:53:89 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::ff:fe52:5389/64 scope link valid_lft forever preferred_lft forever Complete procedure to reproduce the issue ========================================= It is here assumed that there is a DHCP server available elsewhere on the network. Set-up ------ 1. Install an amd64 Debian Buster (default network install), 2. create a bridge on the host with a static IP and deactivate DHCP, in `/etc/network/interfaces`, ``` # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback ## The primary network interface #allow-hotplug ens18 #iface ens18 inet dhcp ## This is an autoconfigured IPv6 interface #iface ens18 inet6 auto iface ens18 inet manual auto br0 iface br0 inet static address 192.168.1.168 netmask 255.255.255.0 gateway 192.168.1.220 bridge_ports ens18 ``` 3. reboot the host, ```bash reboot ``` 4. install lxc and create a bionic amd64 container, ```bash apt install lxc lxc-create -t download -n bionic ``` 5. on the host, modify the network configuration of the container to use the bridge with a static IP in `/var/lib/lxc/bionic/config`, ``` # Template used to create this container: /usr/share/lxc/templates/lxc-download # Parameters passed to the template: # Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3 # For additional config options, please look at lxc.container.conf(5) # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf # For Ubuntu 14.04 lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0 lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0 lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0 lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0 lxc.arch = linux64 # Container specific configuration lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 lxc.rootfs.path = dir:/var/lib/lxc/bionic/rootfs lxc.uts.name = bionic ## Network configuration #lxc.net.0.type = empty # Network configuration lxc.net.0.type = veth lxc.net.0.flags = up lxc.net.0.link = br0 lxc.net.0.name = eth0 lxc.net.0.ipv4.gateway = 192.168.1.220 lxc.net.0.ipv4.address = 192.168.1.169/32 ``` 6. inside the container, install the systemd packages without the bug, and deactivate DHCP in `/etc/netplan/10-lxc.yaml`, ```bash lxc-start -n bionic lxc-attach -n bionic apt install systemd=237-3ubuntu10.38 libsystemd0=237-3ubuntu10.38 libnss-systemd=237-3ubuntu10.38 libpam-systemd=237-3ubuntu10.38 sed -i 's/true/false/' /etc/netplan/10-lxc.yaml exit ``` 7. stop the container. ```bash lxc-stop -n bionic ``` Let’s do it ----------- 1. Start the container and check the IP config, which should be ok, ```bash lxc-start -n bionic lxc-attach -n bionic ip a ``` 2. upgrade the system and check the IP config, the static IP is gone. ```bash apt upgrade ip a exit ``` If systemd is downgraded again to 237-3ubuntu10.38, the IP is back at the next reboot of the container. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1863873/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
