Public bug reported: * High level description:
I've been attempting to enable Fedora 32 support in devstack and encountered the following issue where dnsmasq as configured by q-dhcp isn't responding to DHCP requests from clients: https://review.opendev.org/#/c/750292/ Looking at tcpdump and strace it appears that dnsmasq can see the requests but doesn't reply suggesting a configuration issue either caused by q-dhcp *or* a regression in dnsmasq itself: $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes 18:40:24.070796 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 18:41:24.118961 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 18:42:24.192716 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 strace -p 196856 strace: Process 196856 attached restart_syscall(<... resuming interrupted read ...>) = 1 recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 300 recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, 0) = 300 ioctl(4, SIOCGIFNAME, {ifr_index=9, ifr_name="tapee679459-e1"}) = 0 ioctl(4, SIOCGIFFLAGS, {ifr_name="tapee679459-e1", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0 ioctl(4, SIOCGIFADDR, {ifr_name="tapee679459-e1", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.0.0.2")}}) = 0 poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=14, events=POLLIN}], 8, -1 The current configs are listed below: http://paste.openstack.org/show/798334/ I was able to downgrade dnsmasq on f32 to 2.80 in order to workaround this: $ sudo dnf downgrade dnsmasq -y [..] $ rpm -qa | grep dnsmasq dnsmasq-2.80-14.fc32.x86_64 $ sudo killall dnsmasq && sudo systemctl restart devstack@q-* $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes 12:06:57.028953 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 12:06:57.029994 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 328 12:06:57.042300 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 12:06:57.047014 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 344 * Pre-conditions: F32 with dnsmasq >= 2.81 installed. * Step-by-step reproduction steps: Deploy F32 with dnsmasq >= 2.81 installed, attempt to spawn an instance attached to a subnet with dhcp enabled. * Expected output: dnsmasq responds to DHCP request from instance. * Actual output: dnsmasq doesn't respond to DHCP request from instance. * Version: ** OpenStack version (Specific stable branch, or git hash if from trunk); Neutron @ 0fdcc4b1b63dc90fbc9f46f5947f84626f8e5b41 ** Linux distro, kernel. For a distro, it’s also worth knowing specific versions of client and server; Fedora 32 with kernel 5.8.10-200.fc32.x86_64 ** DevStack or other _deployment_ mechanism? Devstack @ https://review.opendev.org/#/c/750292/ * Environment: what types of services are you running (core services like DB and AMQP broker, as well as Nova/hypervisor if it matters), and which type of deployment (clustered servers)? Multi-node or single node, etc. Single node devstack env. * Perceived severity: is this a blocker for you? High, assuming other distros will be impacted by this once they move to dnsmasq >= 2.81 ** Affects: neutron Importance: Undecided Status: New ** Tags: l3-ipam-dhcp -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1896945 Title: dnsmasq >= 2.81 not responding to DHCP requests with current q-dhcp configs Status in neutron: New Bug description: * High level description: I've been attempting to enable Fedora 32 support in devstack and encountered the following issue where dnsmasq as configured by q-dhcp isn't responding to DHCP requests from clients: https://review.opendev.org/#/c/750292/ Looking at tcpdump and strace it appears that dnsmasq can see the requests but doesn't reply suggesting a configuration issue either caused by q-dhcp *or* a regression in dnsmasq itself: $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes 18:40:24.070796 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 18:41:24.118961 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 18:42:24.192716 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 strace -p 196856 strace: Process 196856 attached restart_syscall(<... resuming interrupted read ...>) = 1 recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, MSG_PEEK|MSG_TRUNC) = 300 recvmsg(4, {msg_name={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, msg_namelen=16, msg_iov=[{iov_base="\1\1\6\0\0041\326S\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\26"..., iov_len=548}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("tapee679459-e1"), ipi_spec_dst=inet_addr("10.0.0.2"), ipi_addr=inet_addr("255.255.255.255")}}], msg_controllen=32, msg_flags=0}, 0) = 300 ioctl(4, SIOCGIFNAME, {ifr_index=9, ifr_name="tapee679459-e1"}) = 0 ioctl(4, SIOCGIFFLAGS, {ifr_name="tapee679459-e1", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0 ioctl(4, SIOCGIFADDR, {ifr_name="tapee679459-e1", ifr_addr={sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.0.0.2")}}) = 0 poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}, {fd=14, events=POLLIN}], 8, -1 The current configs are listed below: http://paste.openstack.org/show/798334/ I was able to downgrade dnsmasq on f32 to 2.80 in order to workaround this: $ sudo dnf downgrade dnsmasq -y [..] $ rpm -qa | grep dnsmasq dnsmasq-2.80-14.fc32.x86_64 $ sudo killall dnsmasq && sudo systemctl restart devstack@q-* $ openstack server reboot --hard test && sudo ip netns exec qdhcp-df64061e-0784-4bbe-909b-ae1c5f466981 tcpdump -i tapee679459-e1 -n port 67 or port 68 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapee679459-e1, link-type EN10MB (Ethernet), capture size 262144 bytes 12:06:57.028953 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 12:06:57.029994 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 328 12:06:57.042300 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:16:07:a0, length 300 12:06:57.047014 IP 10.0.0.2.bootps > 10.0.0.49.bootpc: BOOTP/DHCP, Reply, length 344 * Pre-conditions: F32 with dnsmasq >= 2.81 installed. * Step-by-step reproduction steps: Deploy F32 with dnsmasq >= 2.81 installed, attempt to spawn an instance attached to a subnet with dhcp enabled. * Expected output: dnsmasq responds to DHCP request from instance. * Actual output: dnsmasq doesn't respond to DHCP request from instance. * Version: ** OpenStack version (Specific stable branch, or git hash if from trunk); Neutron @ 0fdcc4b1b63dc90fbc9f46f5947f84626f8e5b41 ** Linux distro, kernel. For a distro, it’s also worth knowing specific versions of client and server; Fedora 32 with kernel 5.8.10-200.fc32.x86_64 ** DevStack or other _deployment_ mechanism? Devstack @ https://review.opendev.org/#/c/750292/ * Environment: what types of services are you running (core services like DB and AMQP broker, as well as Nova/hypervisor if it matters), and which type of deployment (clustered servers)? Multi-node or single node, etc. Single node devstack env. * Perceived severity: is this a blocker for you? High, assuming other distros will be impacted by this once they move to dnsmasq >= 2.81 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1896945/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
