On Wed, 12 Feb 2020 at 12:47, Yedidyah Bar David <[email protected]> wrote:
> On Wed, Feb 12, 2020 at 12:27 PM Yedidyah Bar David <[email protected]> > wrote: > >> On Mon, Feb 10, 2020 at 3:19 PM Barak Korren <[email protected]> wrote: >> >>> >>> >>> On Mon, 10 Feb 2020 at 15:11, Dominik Holler <[email protected]> wrote: >>> >>>> >>>> On Mon, Feb 10, 2020 at 10:32 AM Dominik Holler <[email protected]> >>>> wrote: >>>> >>>>> Hello, >>>>> is this issue reproducible local for someone, or does this happen only >>>>> on jenkins? >>>>> >>>>> >>>> For me it seems to happen only on jenkins. >>>> I don't know enough about the environment on jenkins. >>>> Might a "yum update" fix the problem? >>>> >>> >>> >>> `yum update` actually caused the problem... >>> >>> >>>> I guess that there are incompatible versions of libvirt and firewalld >>>> installed. >>>> >>> >>> We have the latest versions for both, currently we suspect its the >>> startup order >>> >>> >>>> >>>> >>>>> On Mon, Feb 10, 2020 at 9:46 AM Galit Rosenthal <[email protected]> >>>>> wrote: >>>>> >>>>>> Checking this >>>>>> Once I have more info I will update. >>>>>> >>>>>> On Mon, Feb 10, 2020 at 9:16 AM Parth Dhanjal <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey! >>>>>>> >>>>>>> hc_basic_suite_4.3 is failing with the same error as well >>>>>>> >>>>>>> Project: >>>>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_hc-basic-suite-4.3/ >>>>>>> Build: >>>>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_hc-basic-suite-4.3/328/consoleFul >>>>>>> <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_hc-basic-suite-4.3/328/consoleFull> >>>>>>> l >>>>>>> >>>>>>> *07:36:24* @ Start Prefix: *07:36:24* # Start nets: *07:36:24* * >>>>>>> Create network lago-hc-basic-suite-4-3-net-management: *07:36:30* * >>>>>>> Create network lago-hc-basic-suite-4-3-net-management: ERROR (in >>>>>>> 0:00:05)*07:36:30* # Start nets: ERROR (in 0:00:05)*07:36:30* @ Start >>>>>>> Prefix: ERROR (in 0:00:05)*07:36:30* Error occured, aborting*07:36:30* >>>>>>> Traceback (most recent call last):*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/cmd.py", line 969, in >>>>>>> main*07:36:30* cli_plugins[args.verb].do_run(args)*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in >>>>>>> do_run*07:36:30* self._do_run(**vars(args))*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in >>>>>>> wrapper*07:36:30* return func(*args, **kwargs)*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in >>>>>>> wrapper*07:36:30* return func(*args, prefix=prefix, >>>>>>> **kwargs)*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/cmd.py", line 271, in >>>>>>> do_start*07:36:30* prefix.start(vm_names=vm_names)*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in >>>>>>> wrapped*07:36:30* return func(*args, **kwargs)*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1323, in >>>>>>> start*07:36:30* self.virt_env.start(vm_names=vm_names)*07:36:30* >>>>>>> File "/usr/lib/python2.7/site-packages/lago/virt.py", line 341, in >>>>>>> start*07:36:30* net.start()*07:36:30* File >>>>>>> "/usr/lib/python2.7/site-packages/lago/providers/libvirt/network.py", >>>>>>> line 115, in start*07:36:30* net = >>>>>>> self.libvirt_con.networkCreateXML(self._libvirt_xml())*07:36:30* File >>>>>>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 4216, in >>>>>>> networkCreateXML*07:36:30* if ret is None:raise >>>>>>> libvirtError('virNetworkCreateXML() failed', conn=self)*07:36:30* >>>>>>> libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid >>>>>>> backend or is unavailable >>>>>>> >>>>>>> >> I now spent some time and think I at least managed to find what emits >> this error: >> >> https://github.com/firewalld/firewalld/blob/master/src/firewall/core/fw.py >> >> So perhaps it's an update to firewalld, or to iptables, ip6tables, etc. >> > > I think it is: > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/348/consoleFull > > *03:56:02* ---> Package firewalld.noarch 0:0.5.3-5.el7 will be updated > > *03:56:02* ---> Package firewalld.noarch 0:0.6.3-2.el7_7.3 will be an update > > This means we actually have an updated package by the time the error occurs. After some research we reached the conclusion this has to do with having firewalld run and get updated inside a privileged container. It seems that during its startup process it removes the IPv6 kernel modules from the running kernel and then fails to re-load them because the module files are not available from inside the container. Galit is working on a patch to remove firewalld from the container since we don't really need it there. > >> >>> *07:36:30* + on_exit*07:36:30* + [[ 1 -ne 0 ]]*07:36:30* + logger.error >>> 'on_exit: Exiting with a non-zero status'*07:36:30* + logger.log ERROR >>> 'on_exit: Exiting with a non-zero status'*07:36:30* + set +x*07:36:30* >>> 2020-02-10 02:06:30.296966051+0000 run_suite.sh::on_exit::ERROR:: on_exit: >>> Exiting with a non-zero status >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 10, 2020 at 12:08 PM Yedidyah Bar David <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> On Mon, Feb 10, 2020 at 4:13 AM <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Project: >>>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ >>>>>>>> > Build: >>>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/346/ >>>>>>>> >>>>>>>> lago.log has: >>>>>>>> >>>>>>>> 1. some repo issue: >>>>>>>> >>>>>>>> 2020-02-10 02:11:13,681::ERROR::repoman.common.parser::No artifacts >>>>>>>> found for source /var/lib/lago/ovirt-appliance-4.3-el7:only-missing >>>>>>>> >>>>>>>> Galit - any idea? >>>>>>>> >>>>>>>> 2. IPv6 issue - after a long series of tracebacks: >>>>>>>> >>>>>>>> libvirtError: COMMAND_FAILED: INVALID_IPV: 'ipv6' is not a valid >>>>>>>> backend or is unavailable >>>>>>>> >>>>>>>> Dominik, any idea? Not sure if this is an infra issue or a recent >>>>>>>> change to OST (or lago, or libvirt...). >>>>>>>> >>>>>>>> Thanks and best regards, >>>>>>>> -- >>>>>>>> Didi >>>>>>>> _______________________________________________ >>>>>>>> Infra mailing list -- [email protected] >>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/URERUP2VUCW25RFUIOLVEE3DMBMHBF6C/ >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> GALIT ROSENTHAL >>>>>> >>>>>> SOFTWARE ENGINEER >>>>>> >>>>>> Red Hat >>>>>> >>>>>> <https://www.redhat.com/> >>>>>> >>>>>> [email protected] T: 972-9-7692230 >>>>>> <https://red.ht/sig> >>>>>> >>>>> _______________________________________________ >>>> Infra mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/[email protected]/message/SYYL7RVRKNWHD64CIBWGXBV4REOGVHFK/ >>>> >>> >>> >>> -- >>> Barak Korren >>> RHV DevOps team , RHCE, RHCi >>> Red Hat EMEA >>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>> _______________________________________________ >>> Infra mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/[email protected]/message/574TXW4JBJGY53OI5QGAWCZVO6DLO4R2/ >>> >> >> >> -- >> Didi >> > > > -- > Didi > -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________ Infra mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/H3YQHTHC2I6XGJEP7IKNZFYFT6X7P6J6/
