Hi, As Kai Qiang mentioned, magnum gate recently had a bunch of random failures, which occurred on creating a nova instance with 2G of RAM. According to the error message, it seems that the hypervisor tried to allocate memory to the nova instance but couldn’t find enough free memory in the host. However, by adding a few “nova hypervisor-show XX” before, during, and right after the test, it showed that the host has 6G of free RAM, which is far more than 2G. Here is a snapshot of the output [1]. You can find the full log here [2].
Another observation is that most of the failure happened on a node with name “devstack-trusty-ovh-*” (You can verify it by entering a query [3] at http://logstash.openstack.org/ ). It seems that the jobs will be fine if they are allocated to a node other than “ovh”. Any hints to debug this issue further? Suggestions are greatly appreciated. [1] http://paste.openstack.org/show/481746/ [2] http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html [3] https://review.openstack.org/#/c/254370/2/queries/1521237.yaml Best regards, Hongbin From: Kai Qiang Wu [mailto:[email protected]] Sent: December-09-15 7:23 AM To: [email protected] Subject: [openstack-dev] [Infra][nova][magnum] Jenkins failed quite often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory" Hi All, I am not sure what changes these days, We found quite often now, the Jenkins failed for: http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz 2015-12-09 08:52:27.892<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000: 22957: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1 2015-12-09 08:52:27.892<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000: 22957: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG: mon=0x7fa66400c6f0 msg={"execute":"qmp_capabilities","id":"libvirt-1"} fd=-1 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled. 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled. 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6 2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled. 2015-12-09 08:52:28.070<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000: 22951: error : qemuMonitorIORead:554 : Unable to read from monitor: Connection reset by peer 2015-12-09 08:52:28.070<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000: 22951: error : qemuMonitorIO:690 : internal error: early end of file from monitor: possible problem: Cannot set up guest memory 'pc.ram': Cannot allocate memory It not hit such resource issue before. I am not sure if Infra or nova guys know about it ? Thanks Best Wishes, -------------------------------------------------------------------------------- Kai Qiang Wu (吴开强 Kennan） IBM China System and Technology Lab, Beijing E-mail: [email protected]<mailto:[email protected]> Tel: 86-10-82451647 Address: Building 28(Ring Building), ZhongGuanCun Software Park, No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193 -------------------------------------------------------------------------------- Follow your heart. You are miracle!
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
