Public bug reported:
This is a serious regression with systemd-networkd that I ran in to
while setting up a NAT router in AWS. The AWS AMI ubuntu/images/hvm-ssd
/ubuntu-bionic-18.04-amd64-server-20200131 with systemd-237-3ubuntu10.33
does NOT have the problem, but the next most recent AWS AMI
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200311 with
systemd-including 237-3ubuntu10.39 does.
Also, a system booted from the (good) 20200131 AMI starts showing the
problem after updating only systemd (to 237-3ubuntu10.41) and its direct
dependencies (e.g. 'apt-get install systemd'). So I'm fairly confident
that a change to the systemd package between 237-3ubuntu10.33 and
237-3ubuntu10.39 introduced the problem and it is still present.
On the NAT router I use three interfaces and have separate routing
tables for admin and forwarded traffic. Things come up fine initially
but every 30-60 minutes (DHCP lease renewal time?) one or more
interfaces is reconfigured and most of the time systemd-networkd will
crash and need to be restarted. Eventually the system becomes
unreachable when the default crash loop backoff logic prevents the
network service from being restarted at all. The log excerpt attached
illustrates the crash loop.
Also including the netplan and networkd config files below.
# grep . /etc/netplan/*
/etc/netplan/50-cloud-init.yaml:# This file is generated from information
provided by the datasource. Changes
/etc/netplan/50-cloud-init.yaml:# to it will not persist across an instance
reboot. To disable cloud-init's
/etc/netplan/50-cloud-init.yaml:# network configuration capabilities, write a
file
/etc/netplan/50-cloud-init.yaml:#
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
/etc/netplan/50-cloud-init.yaml:# network: {config: disabled}
/etc/netplan/50-cloud-init.yaml:network:
/etc/netplan/50-cloud-init.yaml: version: 2
/etc/netplan/50-cloud-init.yaml: ethernets:
/etc/netplan/50-cloud-init.yaml: ens5:
/etc/netplan/50-cloud-init.yaml: dhcp4: true
/etc/netplan/50-cloud-init.yaml: match:
/etc/netplan/50-cloud-init.yaml: macaddress: xx:xx:xx:xx:xx:xx
/etc/netplan/50-cloud-init.yaml: set-name: ens5
/etc/netplan/99_config.yaml:network:
/etc/netplan/99_config.yaml: version: 2
/etc/netplan/99_config.yaml: renderer: networkd
/etc/netplan/99_config.yaml: ethernets:
/etc/netplan/99_config.yaml: ens6:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: yy:yy:yy:yy:yy:yy
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-routes: false
/etc/netplan/99_config.yaml: ens7:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: zz:zz:zz:zz:zz:zz
/etc/netplan/99_config.yaml: mtu: 1500
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-mtu: false
/etc/netplan/99_config.yaml: use-routes: false
# grep . /etc/networkd-dispatcher/*/*
/etc/networkd-dispatcher/configured.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configured.d/nat:# Do additional configuration for the
inside and outside interfaces
/etc/networkd-dispatcher/configured.d/nat:# route table used for
forwarded/routed/natted traffic
/etc/networkd-dispatcher/configured.d/nat:FWD_TABLE=99
/etc/networkd-dispatcher/configured.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
inside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.3.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for inside
in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace 10.0.3.0/24
dev ens6 scope link src 10.0.3.171 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add routes to VPC cidrs via
inside gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace 10.0.0.0/16
via 10.0.3.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens6 lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens6 lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.3.171/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
outside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.2.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for outside
in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace 10.0.2.0/24
dev ens7 scope link src 10.0.2.245 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add default route via outside
gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace default via
10.0.2.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens7 lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens7 lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.2.245/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use the inet route
for local traffic but only if it's not destined for an RFC1918 private range
/etc/networkd-dispatcher/configured.d/nat: # IMPORTANT: order matters; the
priority of rules is reverse of the order in which they are added.
/etc/networkd-dispatcher/configured.d/nat: # so the default/fallback is added
first and then the local overrides.
/etc/networkd-dispatcher/configured.d/nat: #/sbin/ip rule add iif lo lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 10.0.0.0/8 iif lo
lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 172.16.0.0/12 iif
lo lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 192.168.0.0/16 iif
lo lookup main
/etc/networkd-dispatcher/configured.d/nat: # ensure the forward policy is
accept
/etc/networkd-dispatcher/configured.d/nat: iptables -P FORWARD ACCEPT
/etc/networkd-dispatcher/configured.d/nat: # configure iptables to do NAT
/etc/networkd-dispatcher/configured.d/nat: /sbin/iptables -t nat -I
POSTROUTING 1 -o ens7 -j SNAT --to-source 10.0.2.245
/etc/networkd-dispatcher/configured.d/nat: # clean up any other rules
/etc/networkd-dispatcher/configured.d/nat: while /sbin/iptables -t nat -D
POSTROUTING 2 2>/dev/null; do :; done
/etc/networkd-dispatcher/configured.d/nat:fi
/etc/networkd-dispatcher/configuring.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configuring.d/nat:# Tear down existing ip rules so
they aren't duplicated
/etc/networkd-dispatcher/configuring.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referenceing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens6|10.0.3.171" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referencing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens7|10.0.2.245|iif lo" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:fi
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.39
ProcVersionSignature: Ubuntu 4.15.0-1060.62-aws 4.15.18
Uname: Linux 4.15.0-1060-aws x86_64
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
Date: Wed Jun 3 21:24:28 2020
Ec2AMI: ami-0238c6e72a7e906fc
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: c5n.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Amazon EC2 c5n.large
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1060-aws
root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
nvme_core.io_timeout=4294967295
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/16/2017
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-0c058310742990713
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias:
dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:svnAmazonEC2:pnc5n.large:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:
dmi.product.name: c5n.large
dmi.sys.vendor: Amazon EC2
** Affects: systemd (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug bionic ec2-images
** Attachment added: "systemd-networkd log output with comments"
https://bugs.launchpad.net/bugs/1881972/+attachment/5380157/+files/nat-networkd-log.txt
** Description changed:
This is a serious regression with systemd-networkd that I ran in to
while setting up a NAT router in AWS. The AWS AMI ubuntu/images/hvm-ssd
/ubuntu-bionic-18.04-amd64-server-20200131 with systemd-237-3ubuntu10.33
does NOT have the problem, but the next most recent AWS AMI
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200311 with
systemd-including 237-3ubuntu10.39 does.
Also, a system booted from the (good) 20200131 AMI starts showing the
problem after updating only systemd (to 237-3ubuntu10.41) and its direct
dependencies (e.g. 'apt-get install systemd'). So I'm fairly confident
that a change to the systemd package between 237-3ubuntu10.33 and
237-3ubuntu10.39 introduced the problem and it is still present.
On the NAT router I use three interfaces and have separate routing
tables for admin and forwarded traffic. Things come up fine initially
but every 30-60 minutes (DHCP lease renewal time?) one or more
interfaces is reconfigured and most of the time systemd-networkd will
crash and need to be restarted. Eventually the system becomes
unreachable when the default crash loop backoff logic prevents the
network service from being restarted at all. The log excerpt attached
illustrates the crash loop.
Also including the netplan and networkd config files below.
# grep . /etc/netplan/*
/etc/netplan/50-cloud-init.yaml:# This file is generated from information
provided by the datasource. Changes
/etc/netplan/50-cloud-init.yaml:# to it will not persist across an instance
reboot. To disable cloud-init's
/etc/netplan/50-cloud-init.yaml:# network configuration capabilities, write a
file
/etc/netplan/50-cloud-init.yaml:#
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
/etc/netplan/50-cloud-init.yaml:# network: {config: disabled}
/etc/netplan/50-cloud-init.yaml:network:
/etc/netplan/50-cloud-init.yaml: version: 2
/etc/netplan/50-cloud-init.yaml: ethernets:
/etc/netplan/50-cloud-init.yaml: ens5:
/etc/netplan/50-cloud-init.yaml: dhcp4: true
/etc/netplan/50-cloud-init.yaml: match:
/etc/netplan/50-cloud-init.yaml: macaddress: xx:xx:xx:xx:xx:xx
/etc/netplan/50-cloud-init.yaml: set-name: ens5
/etc/netplan/99_config.yaml:network:
/etc/netplan/99_config.yaml: version: 2
/etc/netplan/99_config.yaml: renderer: networkd
/etc/netplan/99_config.yaml: ethernets:
/etc/netplan/99_config.yaml: ens6:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: yy:yy:yy:yy:yy:yy
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-routes: false
/etc/netplan/99_config.yaml: ens7:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: zz:zz:zz:zz:zz:zz
/etc/netplan/99_config.yaml: mtu: 1500
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-mtu: false
/etc/netplan/99_config.yaml: use-routes: false
- # grep . /etc/networkd-dispatcher/*/*|sed -e "s/domo-nat/nat/g"
+ # grep . /etc/networkd-dispatcher/*/*
/etc/networkd-dispatcher/configured.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configured.d/nat:# Do additional configuration for
the inside and outside interfaces
/etc/networkd-dispatcher/configured.d/nat:# route table used for
forwarded/routed/natted traffic
/etc/networkd-dispatcher/configured.d/nat:FWD_TABLE=99
/etc/networkd-dispatcher/configured.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
inside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.3.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for inside
in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.3.0/24 dev ens6 scope link src 10.0.3.171 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add routes to VPC cidrs via
inside gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.0.0/16 via 10.0.3.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens6
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens6
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.3.171/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
outside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.2.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for
outside in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.2.0/24 dev ens7 scope link src 10.0.2.245 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add default route via outside
gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace default
via 10.0.2.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens7
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens7
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.2.245/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use the inet route
for local traffic but only if it's not destined for an RFC1918 private range
/etc/networkd-dispatcher/configured.d/nat: # IMPORTANT: order matters; the
priority of rules is reverse of the order in which they are added.
/etc/networkd-dispatcher/configured.d/nat: # so the default/fallback is
added first and then the local overrides.
/etc/networkd-dispatcher/configured.d/nat: #/sbin/ip rule add iif lo lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 10.0.0.0/8 iif lo
lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 172.16.0.0/12 iif
lo lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 192.168.0.0/16
iif lo lookup main
/etc/networkd-dispatcher/configured.d/nat: # ensure the forward policy is
accept
/etc/networkd-dispatcher/configured.d/nat: iptables -P FORWARD ACCEPT
/etc/networkd-dispatcher/configured.d/nat: # configure iptables to do NAT
/etc/networkd-dispatcher/configured.d/nat: /sbin/iptables -t nat -I
POSTROUTING 1 -o ens7 -j SNAT --to-source 10.0.2.245
/etc/networkd-dispatcher/configured.d/nat: # clean up any other rules
/etc/networkd-dispatcher/configured.d/nat: while /sbin/iptables -t nat -D
POSTROUTING 2 2>/dev/null; do :; done
/etc/networkd-dispatcher/configured.d/nat:fi
/etc/networkd-dispatcher/configuring.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configuring.d/nat:# Tear down existing ip rules so
they aren't duplicated
/etc/networkd-dispatcher/configuring.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referenceing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens6|10.0.3.171" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referencing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens7|10.0.2.245|iif lo" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:fi
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.39
ProcVersionSignature: Ubuntu 4.15.0-1060.62-aws 4.15.18
Uname: Linux 4.15.0-1060-aws x86_64
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
Date: Wed Jun 3 21:24:28 2020
Ec2AMI: ami-0238c6e72a7e906fc
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: c5n.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Amazon EC2 c5n.large
ProcEnviron:
- TERM=xterm-256color
- PATH=(custom, no user)
- LANG=C.UTF-8
- SHELL=/bin/bash
+ TERM=xterm-256color
+ PATH=(custom, no user)
+ LANG=C.UTF-8
+ SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1060-aws
root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
nvme_core.io_timeout=4294967295
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/16/2017
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-0c058310742990713
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias:
dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:svnAmazonEC2:pnc5n.large:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:
dmi.product.name: c5n.large
dmi.sys.vendor: Amazon EC2
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1881972
Title:
systemd-networkd crashes with invalid pointer
Status in systemd package in Ubuntu:
New
Bug description:
This is a serious regression with systemd-networkd that I ran in to
while setting up a NAT router in AWS. The AWS AMI ubuntu/images/hvm-
ssd/ubuntu-bionic-18.04-amd64-server-20200131 with
systemd-237-3ubuntu10.33 does NOT have the problem, but the next most
recent AWS AMI ubuntu/images/hvm-ssd/ubuntu-
bionic-18.04-amd64-server-20200311 with systemd-including
237-3ubuntu10.39 does.
Also, a system booted from the (good) 20200131 AMI starts showing the
problem after updating only systemd (to 237-3ubuntu10.41) and its
direct dependencies (e.g. 'apt-get install systemd'). So I'm fairly
confident that a change to the systemd package between
237-3ubuntu10.33 and 237-3ubuntu10.39 introduced the problem and it is
still present.
On the NAT router I use three interfaces and have separate routing
tables for admin and forwarded traffic. Things come up fine initially
but every 30-60 minutes (DHCP lease renewal time?) one or more
interfaces is reconfigured and most of the time systemd-networkd will
crash and need to be restarted. Eventually the system becomes
unreachable when the default crash loop backoff logic prevents the
network service from being restarted at all. The log excerpt attached
illustrates the crash loop.
Also including the netplan and networkd config files below.
# grep . /etc/netplan/*
/etc/netplan/50-cloud-init.yaml:# This file is generated from information
provided by the datasource. Changes
/etc/netplan/50-cloud-init.yaml:# to it will not persist across an instance
reboot. To disable cloud-init's
/etc/netplan/50-cloud-init.yaml:# network configuration capabilities, write a
file
/etc/netplan/50-cloud-init.yaml:#
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
/etc/netplan/50-cloud-init.yaml:# network: {config: disabled}
/etc/netplan/50-cloud-init.yaml:network:
/etc/netplan/50-cloud-init.yaml: version: 2
/etc/netplan/50-cloud-init.yaml: ethernets:
/etc/netplan/50-cloud-init.yaml: ens5:
/etc/netplan/50-cloud-init.yaml: dhcp4: true
/etc/netplan/50-cloud-init.yaml: match:
/etc/netplan/50-cloud-init.yaml: macaddress: xx:xx:xx:xx:xx:xx
/etc/netplan/50-cloud-init.yaml: set-name: ens5
/etc/netplan/99_config.yaml:network:
/etc/netplan/99_config.yaml: version: 2
/etc/netplan/99_config.yaml: renderer: networkd
/etc/netplan/99_config.yaml: ethernets:
/etc/netplan/99_config.yaml: ens6:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: yy:yy:yy:yy:yy:yy
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-routes: false
/etc/netplan/99_config.yaml: ens7:
/etc/netplan/99_config.yaml: match:
/etc/netplan/99_config.yaml: macaddress: zz:zz:zz:zz:zz:zz
/etc/netplan/99_config.yaml: mtu: 1500
/etc/netplan/99_config.yaml: dhcp4: true
/etc/netplan/99_config.yaml: dhcp4-overrides:
/etc/netplan/99_config.yaml: use-mtu: false
/etc/netplan/99_config.yaml: use-routes: false
# grep . /etc/networkd-dispatcher/*/*
/etc/networkd-dispatcher/configured.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configured.d/nat:# Do additional configuration for
the inside and outside interfaces
/etc/networkd-dispatcher/configured.d/nat:# route table used for
forwarded/routed/natted traffic
/etc/networkd-dispatcher/configured.d/nat:FWD_TABLE=99
/etc/networkd-dispatcher/configured.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
inside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.3.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for inside
in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.3.0/24 dev ens6 scope link src 10.0.3.171 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add routes to VPC cidrs via
inside gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.0.0/16 via 10.0.3.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens6
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens6
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.3.171/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configured.d/nat: # delete link-local route for
outside in default table
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route delete 10.0.2.0/24
2>/dev/null || true
/etc/networkd-dispatcher/configured.d/nat: # add link-local route for
outside in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace
10.0.2.0/24 dev ens7 scope link src 10.0.2.245 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add default route via outside
gateway in table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip route replace default
via 10.0.2.1 table ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use table 99
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add iif ens7
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add oif ens7
lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: /sbin/ip rule add from
10.0.2.245/32 lookup ${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: # add rules to use the inet route
for local traffic but only if it's not destined for an RFC1918 private range
/etc/networkd-dispatcher/configured.d/nat: # IMPORTANT: order matters; the
priority of rules is reverse of the order in which they are added.
/etc/networkd-dispatcher/configured.d/nat: # so the default/fallback is
added first and then the local overrides.
/etc/networkd-dispatcher/configured.d/nat: #/sbin/ip rule add iif lo lookup
${FWD_TABLE}
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 10.0.0.0/8 iif lo
lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 172.16.0.0/12 iif
lo lookup main
/etc/networkd-dispatcher/configured.d/nat: #ip rule add to 192.168.0.0/16
iif lo lookup main
/etc/networkd-dispatcher/configured.d/nat: # ensure the forward policy is
accept
/etc/networkd-dispatcher/configured.d/nat: iptables -P FORWARD ACCEPT
/etc/networkd-dispatcher/configured.d/nat: # configure iptables to do NAT
/etc/networkd-dispatcher/configured.d/nat: /sbin/iptables -t nat -I
POSTROUTING 1 -o ens7 -j SNAT --to-source 10.0.2.245
/etc/networkd-dispatcher/configured.d/nat: # clean up any other rules
/etc/networkd-dispatcher/configured.d/nat: while /sbin/iptables -t nat -D
POSTROUTING 2 2>/dev/null; do :; done
/etc/networkd-dispatcher/configured.d/nat:fi
/etc/networkd-dispatcher/configuring.d/nat:#!/bin/bash
/etc/networkd-dispatcher/configuring.d/nat:# Tear down existing ip rules so
they aren't duplicated
/etc/networkd-dispatcher/configuring.d/nat:if [ "${IFACE}" = "ens6" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referenceing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens6|10.0.3.171" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:elif [ "${IFACE}" = "ens7" ]; then
/etc/networkd-dispatcher/configuring.d/nat: # flush any existing rules
referencing this interface
/etc/networkd-dispatcher/configuring.d/nat: OLDIFS="${IFS}"
/etc/networkd-dispatcher/configuring.d/nat: IFS="
/etc/networkd-dispatcher/configuring.d/nat:"
/etc/networkd-dispatcher/configuring.d/nat: for rule in `ip rule show|egrep
"ens7|10.0.2.245|iif lo" | cut -d: -f2-`; do
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat: ip rule delete ${rule}
/etc/networkd-dispatcher/configuring.d/nat: done
/etc/networkd-dispatcher/configuring.d/nat: IFS="${OLDIFS}"
/etc/networkd-dispatcher/configuring.d/nat:fi
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.39
ProcVersionSignature: Ubuntu 4.15.0-1060.62-aws 4.15.18
Uname: Linux 4.15.0-1060-aws x86_64
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
Date: Wed Jun 3 21:24:28 2020
Ec2AMI: ami-0238c6e72a7e906fc
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: c5n.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Amazon EC2 c5n.large
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1060-aws
root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
nvme_core.io_timeout=4294967295
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/16/2017
dmi.bios.vendor: Amazon EC2
dmi.bios.version: 1.0
dmi.board.asset.tag: i-0c058310742990713
dmi.board.vendor: Amazon EC2
dmi.chassis.asset.tag: Amazon EC2
dmi.chassis.type: 1
dmi.chassis.vendor: Amazon EC2
dmi.modalias:
dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:svnAmazonEC2:pnc5n.large:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:
dmi.product.name: c5n.large
dmi.sys.vendor: Amazon EC2
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1881972/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
