Adding Evgheni. On Wed, Apr 19, 2017 at 10:01 AM, Nadav Goldin <[email protected]> wrote:
> Hi Milan, sorry for missing this. > > In short, it looks like a libvirt/qemu error, I guess it lays > somewhere in the nested environment the Jenkins slave runs at. I was > able to extract the libvirt log from this specific run, but there is > nothing useful there, except that there was no proper termination. > From reading here[1] it might be related to a load on the hypervisor, > and the timeout configured for libvirt to wait for qemu. Unfortunately > looking at the this[2] thread, it seems that a patch to configure the > timeout never got into libvirt, which leaves us with a default of 30 > seconds, and that might not be enough in our nested environment. I > presume that if the hypervisor which the Jenkins slave runs is highly > loaded, then when we try to start the vdsm_functional_tests_lago VM, > it might take more than 30 seconds for qemu to respond. > > Another indication of this "hypothesis" is that I never seen this > error on OST - which uses bare-metal slaves. > > Evgheni, do we have the load monitoring on the hypervisor that runs > vm0065.workers-phx.ovirt.org? Not sure if we added that eventually. > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=987088 > [2] https://www.redhat.com/archives/libvir-list/2014-January/msg00410.html > > On Mon, Apr 10, 2017 at 10:56 AM, Milan Zamazal <[email protected]> > wrote: > > Hi, > > > > after my Vdsm patch https://gerrit.ovirt.org/75329 in ovirt-4.1 branch > > had been merged, Jenkins check-merged job > > http://jenkins.ovirt.org/job/vdsm_4.1_check-merged-el7-x86_64/173/ > > failed with the following error: > > > > 07:01:21 @ Start specified VMs: > > 07:01:21 # Start nets: > > 07:01:21 * Create network vdsm_functional_tests_lago: > > 07:01:27 * Create network vdsm_functional_tests_lago: Success (in > 0:00:05) > > 07:01:27 # Start nets: Success (in 0:00:05) > > 07:01:27 # Start vms: > > 07:01:27 * Starting VM vdsm_functional_tests_host-el7: > > 07:02:07 libvirt: QEMU Driver error : monitor socket did not show up: > No such file or directory > > 07:02:07 * Starting VM vdsm_functional_tests_host-el7: ERROR (in > 0:00:40) > > 07:02:07 # Start vms: ERROR (in 0:00:40) > > 07:02:07 # Destroy network vdsm_functional_tests_lago: > > 07:02:07 # Destroy network vdsm_functional_tests_lago: ERROR (in > 0:00:00) > > 07:02:07 @ Start specified VMs: ERROR (in 0:00:46) > > 07:02:07 Error occured, aborting > > 07:02:07 Traceback (most recent call last): > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line > 936, in main > > 07:02:07 cli_plugins[args.verb].do_run(args) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", > line 184, in do_run > > 07:02:07 self._do_run(**vars(args)) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/utils.py", > line 495, in wrapper > > 07:02:07 return func(*args, **kwargs) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/utils.py", > line 506, in wrapper > > 07:02:07 return func(*args, prefix=prefix, **kwargs) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/cmd.py", line > 264, in do_start > > 07:02:07 prefix.start(vm_names=vm_names) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/prefix.py", > line 1033, in start > > 07:02:07 self.virt_env.start(vm_names=vm_names) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/virt.py", line > 331, in start > > 07:02:07 vm.start() > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", > line 299, in start > > 07:02:07 return self.provider.start(*args, **kwargs) > > 07:02:07 File "/usr/lib/python2.7/site-packages/lago/vm.py", line > 106, in start > > 07:02:07 dom = self.libvirt_con.createXML(self._libvirt_xml()) > > 07:02:07 File "/usr/lib64/python2.7/site-packages/libvirt.py", line > 3782, in createXML > > 07:02:07 if ret is None:raise libvirtError('virDomainCreateXML() > failed', conn=self) > > 07:02:07 libvirtError: monitor socket did not show up: No such file or > directory > > 07:02:07 Took 210 seconds > > > > The error is apparently unrelated to my patch since: 1. my patch should > > have nothing to do with VM start; 2. Jenkins has run successfully on the > > following patch (https://gerrit.ovirt.org/75321). FWIW, the preceding > > patch (https://gerrit.ovirt.org/75038) has run successfully too. > > > > Do you know what's wrong? > > > > Thanks, > > Milan > > _______________________________________________ > > Infra mailing list > > [email protected] > > http://lists.ovirt.org/mailman/listinfo/infra > _______________________________________________ > Infra mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/infra > -- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
