[Openstack] DCHP Server Stops Responding
Hello All, I am having problems with some instances not receiving DCHPACK from my VM's dhcp server after extended periods of time. These instances are running heavy network, I/O, and RAM loads when the DHCPREQUEST does not receive a response. Note that this only happens to a handful of instances, not all instances running the exact same load. I have printed a relevant section of the syslog below. My current environment is a multi-host flat dhcp network with nova-network running on each node. All servers and VM's are running Ubuntu 12.04 and using kvm/libvirt. I am also using large NFS servers to transfer big files to and from instances across the VM network. In the past, I used Eucalyptus and the same problem would occur. This tends to happen more frequently with more VMs and overall network load. If anyone has any ideas as to why the dhcp server stops responding to only a few instances occasionally please let me know. Thanks, Justin --- /var/log/syslog: Oct 12 07:53:36 instance2 dhclient: DHCPREQUEST of 192.168.2.2 on eth0 to 192.168.2.3 port 67 Oct 12 07:53:36 instance2 dhclient: DHCPACK of 192.168.2.2 from 192.168.2.3 Oct 12 07:53:36 instance2 dhclient: bound to 192.168.2.2 -- renewal in 56 seconds. Oct 12 07:54:32 instance2 dhclient: DHCPREQUEST of 192.168.2.2 on eth0 to 192.168.2.3 port 67 Oct 12 07:55:04 dhclient: last message repeated 4 times Oct 12 07:55:15 instance2 dhclient: DHCPREQUEST of 192.168.2.2 on eth0 to 192.168.2.3 port 67 Oct 12 07:55:28 instance2 dhclient: DHCPREQUEST of 192.168.2.2 on eth0 to 255.255.255.255 port 67 Oct 12 07:55:37 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 Oct 12 07:55:40 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4 Oct 12 07:55:44 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 Oct 12 07:55:53 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 18 Oct 12 07:56:11 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 Oct 12 07:56:25 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 20 Oct 12 07:56:45 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17 Oct 12 07:57:02 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 Oct 12 07:57:14 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 Oct 12 07:57:28 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 Oct 12 07:57:43 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 21 Oct 12 07:58:04 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 Oct 12 07:58:12 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 16 Oct 12 07:58:28 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 Oct 12 07:58:36 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 Oct 12 07:58:48 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 Oct 12 07:59:02 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13 Oct 12 07:59:15 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 Oct 12 07:59:27 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 Oct 12 07:59:39 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 Oct 12 07:59:54 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11 Oct 12 08:00:05 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 Oct 12 08:00:20 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17 Oct 12 08:00:37 instance2 dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 1 Oct 12 08:00:38 instance2 dhclient: No DHCPOFFERS received. Oct 12 08:01:13 instance2 kernel: [119711.712185] nfs: server 192.168.2.253 not responding, still trying Oct 12 08:17:01 instance2 CRON[11515]: (root) CMD ( cd / run-parts --report /etc/cron.hourly) Oct 12 09:17:01 instance2 CRON[11519]: (root) CMD ( cd / run-parts --report /etc/cron.hourly) ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] DCHP Server Stops Responding
On Oct 12, 2012, at 10:33 AM, Justin Hurley justin.hur...@opensgi.com wrote: Hello All, I am having problems with some instances not receiving DCHPACK from my VM's dhcp server after extended periods of time. These instances are running heavy network, I/O, and RAM loads when the DHCPREQUEST does not receive a response. Note that this only happens to a handful of instances, not all instances running the exact same load. I have printed a relevant section of the syslog below. My current environment is a multi-host flat dhcp network with nova-network running on each node. All servers and VM's are running Ubuntu 12.04 and using kvm/libvirt. I am also using large NFS servers to transfer big files to and from instances across the VM network. In the past, I used Eucalyptus and the same problem would occur. This tends to happen more frequently with more VMs and overall network load. If anyone has any ideas as to why the dhcp server stops responding to only a few instances occasionally please let me know. I haven't seen this before, but you may be able to work around the issue by setting much longer leases: force_dhcp_release=True dhcp_lease_time=86400 # 1 day leases fixed_ip_disassociate_timout=172800 # 2 day timeout Vish ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
