On Sun, Nov 8, 2020 at 6:08 AM <[email protected]> wrote: > > Project: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/561/ > Build Number: 561 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #561 > [Marcin Sobczyk] basic: Report start times and runnings time of tests > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_run_vms > > Error Message: > Error: Fault reason is "Operation Failed". Fault detail is "[Network error > during communication with the Host.]". HTTP response code is 400.
06:05:28 ../basic-suite-master/test-scenarios/004_basic_sanity.py::test_run_vms started at 04:05:28 FAILED (30s) 06:06:00 06:06:00 =================================== FAILURES =================================== https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/561/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log 2020-11-07 23:05:57,825-05 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-2) [06bc4dc1-f61e-4004-8623-8e91e2e2db24] Command 'org.ovirt.engine.core.bll.RunVmOnceCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Connection timeout for host 'lago-basic-suite-master-host-1', last response arrived 23158 ms ago. (Failed with error VDS_NETWORK_ERROR and code 5022) Logs on host-1 seem ok to me, but finish a bit before this, so perhaps indeed we lost communication with it (temporarily?): https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/561/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/log/vdsm/vdsm.log Last line is: 2020-11-07 23:05:04,434-0500 DEBUG (jsonrpc/6) [jsonrpc.JsonRpcServer] Return 'Host.getStats' in bridge with (suppressed) (__init__:360) https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/561/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/log/messages/*view* Last line is: Nov 7 23:04:10 lago-basic-suite-master-host-1 systemd[1]: Started Session c177 of user root. This run (561) isn't the last one - there were a few successful runs after it, and the following (and last) failed one, 565, is due to virt-sparsify (see other thread). But this also happened to me now: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/7566/ Not copying/pasting again - it looks very similar. engine.log has same error, vdsm.log looks ok but finishes a bit earlier. https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/7566/artifact/exported-artifacts/test_logs/basic-suite-master/lago_logs/lago.log 2020-11-08 20:51:57,693::log_utils.py::__exit__::611::lago.prefix::INFO:: # Deploy environment: [32mSuccess [0m (in 0:14:33) 2020-11-08 20:51:57,957::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ Deploy oVirt environment: [32mSuccess [0m (in 0:14:33) 2020-11-08 21:32:41,201::log_utils.py::__enter__::600::lago.prefix::INFO::@ Collect artifacts: [0m [0m 2020-11-08 21:32:41,295::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-1] lago-basic-suite-master-host-0: [0m [0m 2020-11-08 21:32:41,296::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-2] lago-basic-suite-master-host-1: [0m [0m 2020-11-08 21:32:41,297::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-3] lago-basic-suite-master-engine: [0m [0m 2020-11-08 21:32:41,298::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0: failed extracting files: Unable to extract paths from lago-basic-suite-master-host-0: unreachable with SSH 2020-11-08 21:32:41,299::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-0: attempting to extract files with libguestfs 2020-11-08 21:32:41,507::vm.py::extract_paths::402::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1: failed extracting files: Unable to extract paths from lago-basic-suite-master-host-1: unreachable with SSH 2020-11-08 21:32:41,508::vm.py::extract_paths_dead::437::lago.providers.libvirt.vm::DEBUG::lago-basic-suite-master-host-1: attempting to extract files with libguestfs So it seems to me like an infra issue - also lago failed to ssh there (and also to host-0). Can anyone have a look please? Thanks. Best regards, > > Stack Trace: > assert_vm_is_alive = <function is_alive at 0x7fc7d597b488> > engine_api = <ovirtsdk4.Connection object at 0x7fc7d5979a50> > management_gw_ip = '192.168.201.1' > > @order_by(_TEST_LIST) > def test_run_vms(assert_vm_is_alive, engine_api, management_gw_ip): > engine = engine_api.system_service() > > vm_params = types.Vm( > initialization=types.Initialization( > user_name=VM_USER_NAME, > root_password=VM_PASSWORD > ) > ) > > vm_params.initialization.host_name = BACKUP_VM_NAME > backup_vm_service = test_utils.get_vm_service(engine, BACKUP_VM_NAME) > backup_vm_service.start(use_cloud_init=True, vm=vm_params) > > vm_params.initialization.host_name = VM2_NAME > vm2_service = test_utils.get_vm_service(engine, VM2_NAME) > vm2_service.start(use_cloud_init=True, vm=vm_params) > > # CirrOS cloud-init is different, networking doesn't work since it > doesn't support the format oVirt is using > vm_params.initialization.host_name = VM0_NAME # hostname seems to > work, the others not > vm_params.initialization.dns_search = 'lago.local' > vm_params.initialization.domain = 'lago.local' > vm_params.initialization.dns_servers = management_gw_ip > vm0_service = test_utils.get_vm_service(engine, VM0_NAME) > > vm0_service.start(use_cloud_init=True, vm=vm_params) > > ../basic-suite-master/test-scenarios/004_basic_sanity.py:754: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > /usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:32520: in start > return self._internal_action(action, 'start', None, headers, query, wait) > /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in > _internal_action > return future.wait() if wait else future > /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait > return self._code(response) > /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback > self._check_fault(response) > /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault > self._raise_error(response, body.fault) > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > response = <ovirtsdk4.http.Response object at 0x7fc7d6aefad0> > detail = <ovirtsdk4.types.Fault object at 0x7fc7d6aefb50> > > @staticmethod > def _raise_error(response, detail=None): > """ > Creates and raises an error containing the details of the given HTTP > response and fault. > > This method is intended for internal use by other components of the > SDK. Refrain from using it directly, as backwards compatibility isn't > guaranteed. > """ > fault = detail if isinstance(detail, types.Fault) else None > > msg = '' > if fault: > if fault.reason: > if msg: > msg += ' ' > msg = msg + 'Fault reason is "%s".' % fault.reason > if fault.detail: > if msg: > msg += ' ' > msg = msg + 'Fault detail is "%s".' % fault.detail > if response: > if response.code: > if msg: > msg += ' ' > msg = msg + 'HTTP response code is %s.' % response.code > if response.message: > if msg: > msg += ' ' > msg = msg + 'HTTP response message is "%s".' % > response.message > > if isinstance(detail, six.string_types): > if msg: > msg += ' ' > msg = msg + detail + '.' > > class_ = Error > if response is not None: > if response.code in [401, 403]: > class_ = AuthError > elif response.code == 404: > class_ = NotFoundError > > error = class_(msg) > error.code = response.code if response else None > error.fault = fault > > raise error > E Error: Fault reason is "Operation Failed". Fault detail is "[Network > error during communication with the Host.]". HTTP response code is 400. > > /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118: > Error_______________________________________________ > Infra mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/[email protected]/message/D5SHAOBCVBEPZ5OLXL3GV2L2D7GXTRFM/ -- Didi _______________________________________________ Infra mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/ROEHMBETMMIEE5PSQOC7CYBGZSK2TIMN/
