Public bug reported: ---Problem Description--- Firewall blocks dhcp hand shakes of guest through virbr0 and guest fails to get IP address,
During guest boot up, dhcp hand shakes happens between guest and dnsmasq of host via virtual bridge (virbr0) and this is blocked by firewalld causing failure of guest boot. Even dhclient from inside guest fails to retrieve ipaddress due to this issue. Contact Information = Balamuruhan S / [email protected] System Configuration: Guest Kernel: # uname -a Linux ubuntu1810 4.18.0-11-generic #12-Ubuntu SMP Tue Oct 23 19:20:58 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Host Kernel # uname -a Linux ltc-boston128 4.18.0-11-generic #12-Ubuntu SMP Tue Oct 23 19:20:58 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Qemu # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu4 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu3 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu-block-extra:ppc64el 1:2.12+dfsg-3ubuntu8.1 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.12+dfsg-3ubuntu8.1 ppc64el QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.12+dfsg-3ubuntu8.1 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-data 1:2.12+dfsg-3ubuntu8.1 all QEMU full system emulation (data files) ii qemu-system-gui 1:2.12+dfsg-3ubuntu8.1 ppc64el QEMU full system emulation binaries (user interface and audio support) ii qemu-system-ppc 1:2.12+dfsg-3ubuntu8.1 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.12+dfsg-3ubuntu8.1 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii gir1.2-libvirt-glib-1.0:ppc64el 1.0.0-1 ppc64el GObject introspection files for the libvirt-glib library ii gir1.2-libvirt-sandbox-1.0 0.5.1+git20160404-1 ppc64el GObject introspection files for the libvirt-sandbox library ii libvirt-clients 4.6.0-2ubuntu3 ppc64el Programs for the libvirt library ii libvirt-daemon 4.6.0-2ubuntu3 ppc64el Virtualization daemon ii libvirt-daemon-driver-storage-gluster 4.6.0-2ubuntu3 ppc64el Virtualization daemon glusterfs storage driver ii libvirt-daemon-driver-storage-rbd 4.6.0-2ubuntu3 ppc64el Virtualization daemon RBD storage driver ii libvirt-daemon-driver-storage-sheepdog 4.6.0-2ubuntu3 ppc64el Virtualization daemon Sheedog storage driver ii libvirt-daemon-driver-storage-zfs 4.6.0-2ubuntu3 ppc64el Virtualization daemon ZFS storage driver ii libvirt-daemon-system 4.6.0-2ubuntu3 ppc64el Libvirt daemon configuration files ii libvirt-dbus 1.2.0-1 ppc64el libvirt D-Bus API bindings ii libvirt-dev:ppc64el 4.6.0-2ubuntu3 ppc64el development files for the libvirt library ii libvirt-doc 4.6.0-2ubuntu3 all documentation for the libvirt library ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library ii libvirt-glib-1.0-dev:ppc64el 1.0.0-1 ppc64el Development files for the libvirt-glib library ii libvirt-ocaml 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (runtime) ii libvirt-ocaml-dev 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (development files) ii libvirt-sandbox-1.0-5 0.5.1+git20160404-1 ppc64el Application sandbox toolkit shared library ii libvirt-sandbox-1.0-dev 0.5.1+git20160404-1 ppc64el Development files for libvirt-sandbox library ii libvirt-sanlock 4.6.0-2ubuntu3 ppc64el Sanlock plugin for virtlockd ii libvirt-wireshark 4.6.0-2ubuntu3 ppc64el Wireshark dissector for the libvirt protocol ii libvirt0:ppc64el 4.6.0-2ubuntu3 ppc64el library for interfacing with different virtualization systems ii libvirtodbc0 6.1.6+repack-0ubuntu9 ppc64el high-performance database - ODBC libraries ii libvirtualpg-dev:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG development files ii libvirtualpg0:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG shared library ii python-libvirt 4.6.0-1 ppc64el libvirt Python bindings network configuration: # virsh net-dumpxml default <network> <name>default</name> <uuid>097f097b-bcfa-4752-8f2a-e5337be919ce</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:74:2d:65'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> # virsh net-list --all Name State Autostart Persistent ---------------------------------------------------------- default active yes yes Guest xml have , <interface type='bridge'> <mac address='52:54:00:d8:d9:da'/> <source bridge='virbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </interface> Machine Type = Boston ---Debugger--- A debugger is not configured ---Steps to Reproduce--- 1. Firewall configuration on host # firewall-cmd --list-all public target: default icmp-block-inversion: no interfaces: sources: services: dhcpv6-client dns http https libvirt nfs ssh ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: 2. iptables configuration on Host # iptables -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT 3. Boot a healthy Ubuntu 1810 guest and we can observe network config failure from boot log ``` [ OK ] Started Dispatcher daemon for systemd-networkd. [ OK ] Started LXD - container startup/shutdown. [FAILED] Failed to start Wait for Network to be Configured. See 'systemctl status systemd-networkd-wait-online.service' for details. [ OK ] Reached target Network is Online. Starting OpenBSD Secure Shell server... ``` 4. check the tcpdump on virbr0 interface where guest initiates dhcp handshake but no response as firewall blocks ``` # tcpdump -i virbr0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on virbr0, link-type EN10MB (Ethernet), capture size 262144 bytes 05:43:15.487103 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:43:16.090888 IP6 :: > ff02::1:ff9f:a0a1: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe9f:a0a1, length 32 05:43:17.115003 IP6 fe80::5054:ff:fe9f:a0a1 > ip6-allrouters: ICMP6, router solicitation, length 16 05:43:18.505278 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:43:20.735608 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:43:20.833270 IP6 fe80::5054:ff:fe9f:a0a1 > ip6-allrouters: ICMP6, router solicitation, length 16 05:43:25.822006 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:43:28.525650 IP6 fe80::5054:ff:fe9f:a0a1 > ip6-allrouters: ICMP6, router solicitation, length 16 05:43:33.249590 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:43:44.572320 IP6 fe80::5054:ff:fe9f:a0a1 > ip6-allrouters: ICMP6, router solicitation, length 16 05:43:48.996575 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 ``` 5. If we permit dhcp service using firewall-cmd on host then dhcp request from guest is honoured and guest gets IP address, # firewall-cmd --add-service dhcp --zone=public --permanent success # firewall-cmd --reload success # firewall-cmd --list-all public target: default icmp-block-inversion: no interfaces: sources: services: dhcp dhcpv6-client dns http https libvirt nfs ssh ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: 6. tcpdump after permitting firewall to allow dhcp # tcpdump -i virbr0 05:49:40.393979 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 292 05:49:40.394716 IP ltc-boston128.bootps > 192.168.122.75.bootpc: BOOTP/DHCP, Reply, length 300 05:49:40.396966 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:9f:a0:a1 (oui Unknown), length 304 05:49:40.397110 IP ltc-boston128.bootps > 192.168.122.75.bootpc: BOOTP/DHCP, Reply, length 304 05:49:40.402132 ARP, Request who-has ltc-boston128 tell 192.168.122.75, length 28 05:49:40.402148 ARP, Reply ltc-boston128 is-at 52:54:00:74:2d:65 (oui Unknown), length 28 05:49:40.402233 IP 192.168.122.75.46969 > ltc-boston128.domain: 65305+ [1au] A? ntp.ubuntu.com. (43) 05:49:40.402330 IP 192.168.122.75.40065 > ltc-boston128.domain: 41703+ [1au] AAAA? ntp.ubuntu.com. (43) 05:49:45.650940 ARP, Request who-has 192.168.122.75 tell ltc-boston128, length 28 05:49:45.652551 ARP, Reply 192.168.122.75 is-at 52:54:00:9f:a0:a1 (oui Unknown), length 28 05:49:50.658823 IP 192.168.122.75.58061 > chilipepper.canonical.com.ntp: NTPv4, Client, length 48 05:49:50.658855 IP ltc-boston128 > 192.168.122.75: ICMP host chilipepper.canonical.com unreachable - admin prohibited filter, length 84 05:50:00.908379 IP 192.168.122.75.49333 > pugot.canonical.com.ntp: NTPv4, Client, length 48 05:50:00.908411 IP ltc-boston128 > 192.168.122.75: ICMP host pugot.canonical.com unreachable - admin prohibited filter, length 84 05:50:05.962252 ARP, Request who-has ltc-boston128 tell 192.168.122.75, length 28 05:50:05.962266 ARP, Reply ltc-boston128 is-at 52:54:00:74:2d:65 (oui Unknown), length 28 05:50:11.158012 IP 192.168.122.75.40008 > alphyn.canonical.com.ntp: NTPv4, Client, length 48 05:50:11.158042 IP ltc-boston128 > 192.168.122.75: ICMP host alphyn.canonical.com unreachable - admin prohibited filter, length 84 05:50:16.370934 ARP, Request who-has 192.168.122.75 tell ltc-boston128, length 28 05:50:16.373004 ARP, Reply 192.168.122.75 is-at 52:54:00:9f:a0:a1 (oui Unknown), length 28 Got information that Ubuntu have switched to nftables backend may be causing this issue, but ideally we do not need to add dhcp services in firewall for guest to get IP address Attachment: 1. Sosreport 2. Guest XML Userspace tool common name: firewall The userspace tool has the following bit modes: ppc64le Userspace rpm: # dpkg -l | grep firewall ii firewalld 0.6.3-1 all dynamically managed firewall with support for network zones ii ufw 0.35-6 all program for managing a Netfilter firewall ** Affects: core-network (Ubuntu) Importance: Undecided Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) Status: New ** Tags: architecture-ppc64le bugnameltc-173725 severity-high targetmilestone-inin--- ** Tags added: architecture-ppc64le bugnameltc-173725 severity-high targetmilestone-inin--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1806593 Title: Firewall blocks dhcp hand shakes of guest through virbr0 and guest fails to get IP address, To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/core-network/+bug/1806593/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
