On Tue, Feb 27, 2018 at 2:15 PM, Sandro Bonazzola <sbona...@redhat.com> wrote:
> > > 2018-02-22 22:42 GMT+01:00 oVirt Jenkins <jenk...@ovirt.org>: > >> A system test invoked by the "ovirt-master" change queue including change >> 87428,2 (vdsm) failed. However, this change seems not to be the root >> cause for >> this failure. Change 87944,3 (vdsm) that this change depends on or is >> based on, >> was detected as the cause of the testing failures. >> >> This change had been removed from the testing queue. Artifacts built from >> this >> change will not be released until either change 87944,3 (vdsm) is fixed >> and >> this change is updated to refer to or rebased on the fixed version, or >> this >> change is modified to no longer depend on it. >> >> For further details about the change see: >> https://gerrit.ovirt.org/#/c/87428/2 >> >> For further details about the change that seems to be the root cause >> behind the >> testing failures see: >> https://gerrit.ovirt.org/#/c/87944/3 >> >> For failed test results see: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5841/ > > > > This fails due to multiple reason. > http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/3224/ > failed on > > *22:23:15* E OSError: [Errno 24] Too many open files > > which seems to be an infra issue on the slave / dirty slave > Just FYI, check-merged jobs has nothing to do with OST/CQ results, they are running functional tests written by VDSM developers. > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue- > tester/5841/testReport/junit/(root)/002_bootstrap/add_hosts/ > > Host lago-upgrade-from-release-suite-master-host0 is in non responsive state > > > The host has vdsm failures with: > > 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [MOM] MOM's RPC interface is > disabled (momIF:50) > 2018-02-22 16:34:04,208-0500 ERROR (MainThread) [vds] failed to init > clientIF, shutting down storage dispatcher (clientIF:148) > 2018-02-22 16:34:04,208-0500 INFO (MainThread) [vdsm.api] START > prepareForShutdown(options=None) from=internal, > task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:46) > 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.Monitor] Shutting > down domain monitors (monitor:222) > 2018-02-22 16:34:04,220-0500 INFO (MainThread) [storage.check] Stopping > check service (check:104) > 2018-02-22 16:34:04,221-0500 INFO (check/loop) [storage.asyncevent] Stopping > <EventLoop running=True closed=False at 0x38225104> (asyncevent:220) > 2018-02-22 16:34:04,221-0500 INFO (MainThread) [storage.udev] Stopping > multipath event listener (udev:149) > 2018-02-22 16:34:04,221-0500 INFO (MainThread) [vdsm.api] FINISH > prepareForShutdown return=None from=internal, > task_id=f45a0864-9bd6-4116-add9-a55a05d72909 (api:52) > 2018-02-22 16:34:04,222-0500 ERROR (MainThread) [vds] Exception raised > (vdsmd:158) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 156, in run > serve_clients(log) > File "/usr/lib/python2.7/site-packages/vdsm/vdsmd.py", line 103, in > serve_clients > cif = clientIF.getInstance(irs, log, scheduler) > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 251, in > getInstance > cls._instance = clientIF(irs, log, scheduler) > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 121, in > __init__ > self.mom = MomClient(config.get("mom", "socket_path")) > File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 51, in __init__ > raise MomNotAvailableError() > MomNotAvailableError > > and failure on mom side: > > 2018-02-22 16:34:00,168 - mom - INFO - MOM starting > 2018-02-22 16:34:00,185 - mom.HostMonitor - INFO - Host Monitor starting > 2018-02-22 16:34:00,186 - mom - INFO - hypervisor interface vdsmjsonrpcbulk > 2018-02-22 16:34:00,280 - mom.VdsmRpcBase - ERROR - Cannot connect to VDSM! > [Errno 111] Connection refused > 2018-02-22 16:34:00,280 - mom - ERROR - Failed to initialize MOM threads > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run > hypervisor_iface = self.get_hypervisor_interface() > File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in > get_hypervisor_interface > return module.instance(self.config) > File > "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", > line 47, in instance > return JsonRpcVdsmBulkInterface() > File > "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcbulkInterface.py", > line 29, in __init__ > super(JsonRpcVdsmBulkInterface, self).__init__() > File > "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcInterface.py", > line 41, in __init__ > .orRaise(RuntimeError, 'No connection to VDSM.') > File "/usr/lib/python2.7/site-packages/mom/optional.py", line 28, in orRaise > raise exception(*args, **kwargs) > RuntimeError: No connection to VDSM. > > vdsm upgrade log shows: > > MainThread::DEBUG::2018-02-22 > 16:31:34,113::libvirtconnection::167::root::(get) trying to connect libvirt > MainThread::DEBUG::2018-02-22 16:31:34,134::cmdutils::150::root::(exec_cmd) > lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable > scsi -disable dmi -disable memory -disable cpuinfo (cwd None) > MainThread::DEBUG::2018-02-22 16:31:34,242::cmdutils::158::root::(exec_cmd) > SUCCESS: <err> = ''; <rc> = 0 > MainThread::DEBUG::2018-02-22 16:31:34,269::cmdutils::150::root::(exec_cmd) > /sbin/tc qdisc show (cwd None) > MainThread::DEBUG::2018-02-22 16:31:34,272::cmdutils::158::root::(exec_cmd) > SUCCESS: <err> = ''; <rc> = 0 > MainThread::DEBUG::2018-02-22 16:31:34,273::cmdutils::150::root::(exec_cmd) > /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) > MainThread::DEBUG::2018-02-22 16:31:34,285::cmdutils::158::root::(exec_cmd) > FAILED: <err> = ''; <rc> = 1 > MainThread::INFO::2018-02-22 > 16:31:34,286::netconfpersistence::198::root::(_clearDisk) Clearing netconf: > /var/lib/vdsm/staging/netconf > MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::51::root::(rm_file) > File: /var/lib/vdsm/staging/netconf already removed > MainThread::WARNING::2018-02-22 16:31:34,287::fileutils::96::root::(rm_tree) > Directory: /var/lib/vdsm/staging/netconf already removed > MainThread::INFO::2018-02-22 > 16:31:34,287::netconfpersistence::150::root::(save) Saved new config > RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets and > /var/lib/vdsm/staging/netconf/bonds > MainThread::DEBUG::2018-02-22 16:31:34,288::cmdutils::150::root::(exec_cmd) > /usr/share/openvswitch/scripts/ovs-ctl status (cwd None) > MainThread::DEBUG::2018-02-22 16:31:34,299::cmdutils::158::root::(exec_cmd) > FAILED: <err> = ''; <rc> = 1 > > > Involved versions: > > vdsm-4.30.0-70.git7d8b4e5.el7.centos.x86_64 > > mom-0.5.12-0.0.master.el7.centos.noarch > > > > > > >> >> _______________________________________________ >> Infra mailing list >> Infra@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> > > > > -- > > SANDRO BONAZZOLA > > ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > -- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra