On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <[email protected]> wrote:
> On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <[email protected]> wrote: > > > On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <[email protected]> wrote: > > > > > > I added in > > > https://gerrit.ovirt.org/#/c/98925/ > > > a ping directly before the ssh. > > > The ping succeeds, but the ssh fails. > > > > > > > > > On Tue, 26 Mar 2019 17:07:45 +0100 > > > Sandro Bonazzola <[email protected]> wrote: > > > > > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <[email protected]> > > ha > > > > scritto: > > > > > > > > > +1 from me > > > > > > > > > > > > > Merged. I have 2 patches constantly failing on it, rebased them, you > > can > > > > follow on: > > > > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862 > > > > > > > > > > still failing on jenkins, but at least one succeeds locally for me > > > > Succeeds locally for me also. > > > > Dafna, are we sure there's not an infra issue? > > > > I think since its a race ( and we've seen failures on this test in the > past, also a race I think ), its probably hard to reproduce locally. > Also, we probably need to make sure the same Libvirt version is used. > The upstream servers are quite old, it can also be local run ends up being > faster and not hitting the same issues ( as we've seen in the past ) > > Could it be a bug in the ssh client ( paramiko? ) > Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify? If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection. > Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can > cause this? ( other than slow servers ) > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <[email protected]> > > > > > wrote: > > > > > > > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > > > > > Dominik Holler <[email protected]> wrote: > > > > > > > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > > > > > Dafna Ron <[email protected]> wrote: > > > > > > > > > > > > > > > This is still failing randomly > > > > > > > > > > > > > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to > > understand > > > > > > > which action is crashing the guest. > > > > > > > > > > > > > > > > > > > I was not able to reproduce the failure with the change above. > > > > > > We could merge the change to have better information on the next > > > > > > failure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > > > > > Ryan Barry <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > > > > > > > > > I'm very suspicious of this happening immediately after > > > > > hotplugging a > > > > > > > > > NIC, > > > > > > > > > > especially since the bug attached to > > > > > https://gerrit.ovirt.org/#/c/98765/ > > > > > > > > > > talks about dropping packets. Dominik, did anything else > > change > > > > > here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > > > > > > > > > Is there already a pattern in the failed runs detected, or > > does it > > > > > fail > > > > > > > > > randomly? > > > > > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should > > open a > > > > > bug in > > > > > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > > > > > looking at the logs, we can see that the for some > > reason, > > > > > libvirt > > > > > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > > > > > display="on" for > > > > > > > > > mdevs, > > > > > > > > > > > use nodisplay to override > > > > > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may > > be > > > > > netwrok > > > > > > > > > > > related. > > > > > > > > > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-1/_var_log/libvirt.log > > > > > > > > > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/ > > > > > > > > > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and > > the > > > > > problem is > > > > > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > > > > > qemuDomainObjTaint:7521 : > > > > > > > > > > > Domain id=3 name='vm0' > > > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: > > child > > > > > reported: > > > > > > > > > unable > > > > > > > > > > > to set security context > > 'system_u:object_r:virt_content_t:s0' > > > > > on > > > > > > > > > > > > > > > > > > > > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > > > > > > > No such file or directory > > > > > > > > > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > > > > > files in > > > > > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > > > > > namespace and adding there the devices used by the VM. > > Since > > > > > libvirt > > > > > > > > > > > create the devices in its namespace > > > > > > > > > > > > it should not complain about missing paths in > > > > > /rhev/data-center. > > > > > > > > > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Anton Marchukov > > > > > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > Infra mailing list -- [email protected] > > > > > > > > > > > To unsubscribe send an email to [email protected] > > > > > > > > > > > Privacy Statement: > > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/[email protected]/message/B44Q3AZA7JUPMW4IDWZAS3RYMAFQ56VG/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Devel mailing list -- [email protected] > > > > > > > > > To unsubscribe send an email to [email protected] > > > > > > > > > Privacy Statement: > > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/[email protected]/message/7XYIPXZLPHRRI53QDC24TY6J2ZL2JWSH/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Ryan Barry > > > > > > > > > > Associate Manager - RHV Virt/SLA > > > > > > > > > > [email protected] M: +16518159306 IM: rbarry > > > > > _______________________________________________ > > > > > Infra mailing list -- [email protected] > > > > > To unsubscribe send an email to [email protected] > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > > > https://lists.ovirt.org/archives/list/[email protected]/message/K7WPLV4WFLNURGWBOSISFMAECNC2AXXY/ > > > > > > > > > > > > > > > > > > > > > > -- > > > > Ryan Barry > > > > Associate Manager - RHV Virt/SLA > > > > [email protected] M: +16518159306 IM: rbarry > > _______________________________________________ > > Devel mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/[email protected]/message/J5LSZJVOJV7OSQKIVFNATQTESPXAWQV5/ > > > > _______________________________________________ Devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/NCD6OCOWXSHZYMXWNRA7EQQZVVJBRIUC/
