Il giorno lun 18 feb 2019 alle ore 10:37 Martin Perina <[email protected]> ha scritto:
> > > On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <[email protected]> wrote: > >> Also, try to run the same in mock which would isolate your environment >> > > My environment is minimal CentOS 7 VM installed from scratch, why would > mock help me? > > Unfortunately I have FC29 on laptop and I haven't been able to run OST on > it so far :-( > > I have been discussing with Galit over IRC and so far we haven't been able > to identify any different package version on my CentOS (where everything > works as expected) and her OST environment (where PG 9.5 failed to start). > I guess we'll have to fallback to manual testing for 4.3.1 release since OST doesn't seem to be reliable right now. > >> On Mon, Feb 18, 2019 at 9:28 AM Dafna Ron <[email protected]> wrote: >> >>> are you running upgrade from release or basic? >>> >>> >>> On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <[email protected]> wrote: >>>> >>>>> Martin, >>>>> Is following package also installed in you env? >>>>> >>>>> selinux-policy-targeted-3.13.1-229.el7_6.9.noarch >>>>> >>>>> It is installed in the OST failing environment. >>>>> >>>>> >>>> Yes, both selinux-policy and selinux-policy-targeted are installed and >>>> they didn't cause any issues >>>> >>>>> >>>>> >>>>> Eitan >>>>> >>>>> >>>>> On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm unable to reproduce that issue outside OST, following scenarios >>>>>> worked without any issues: >>>>>> >>>>>> Scenario 1 >>>>>> 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not >>>>>> installed >>>>>> 2. Install and configure ovirt-engine 4.2.8 >>>>>> 3. Login to webadmin - everything works fine >>>>>> 4. Update to selinux-policy-*3.13.1-229.el7_6.9 >>>>>> 5. Login to webadmin - everything works fine >>>>>> 6. Try to restart ovirt-engine and rh-postgresql95-postgresql >>>>>> services >>>>>> 7. Login to webadmin - everything works fine >>>>>> 8. Upgrade all other available packages >>>>>> 9. Login to webadmin - everything works fine >>>>>> 10. Reboot the machine >>>>>> 11. Login to webadmin - everything works fine >>>>>> >>>>>> Senario 2 >>>>>> 1. Update CentOS to latest version and make sure that >>>>>> selinux-policy-*3.13.1-229.el7_6.9 is installed >>>>>> 2. Install and configure ovirt-engine 4.2.8 >>>>>> 3. Login to webadmin - everything works fine >>>>>> >>>>>> So continuing the investigation, but so far it seems to me related >>>>>> only to OST >>>>>> >>>>>> Martin >>>>>> >>>>>> >>>>>> On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Just to add some coal to the fire, here are my findings for failures >>>>>>> of the 4.2 OST network suite: >>>>>>> >>>>>>> Following the selinux update [0], engine setup fails because what >>>>>>> looks like failure of engine to communicate with postgresql. >>>>>>> In [1]: >>>>>>> >>>>>>> Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting >>>>>>> PostgreSQL database server... >>>>>>> Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres >>>>>>> cannot access the server configuration file >>>>>>> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": >>>>>>> Permission denied >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: >>>>>>> could not start server >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine >>>>>>> the log output. >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: >>>>>>> rh-postgresql95-postgresql.service: control process exited, code=exited >>>>>>> status=1 >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start >>>>>>> PostgreSQL database server. >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit >>>>>>> rh-postgresql95-postgresql.service entered failed state. >>>>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: >>>>>>> rh-postgresql95-postgresql.service failed. >>>>>>> >>>>>>> and in [2] there are selinux access denials for pg_ctl to read the >>>>>>> postgres.conf file: >>>>>>> >>>>>>> type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for >>>>>>> pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 >>>>>>> scontext=system_u:system_r:postgresql_t:s0 >>>>>>> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>>>> type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 >>>>>>> success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 >>>>>>> pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 >>>>>>> sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" >>>>>>> exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" >>>>>>> subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>>>> type=PROCTITLE msg=audit(1550363215.978:1067): >>>>>>> proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 >>>>>>> type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for >>>>>>> pid=8648 comm="pg_ctl" >>>>>>> path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" >>>>>>> ino=888709 scontext=system_u:system_r:postgresql_t:s0 >>>>>>> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>>>> type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 >>>>>>> success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 >>>>>>> a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 >>>>>>> gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) >>>>>>> ses=4294967295 comm="pg_ctl" >>>>>>> exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" >>>>>>> subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>>>> type=PROCTITLE msg=audit(1550363215.978:1068): >>>>>>> proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 >>>>>>> type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for >>>>>>> pid=8654 comm="postgres" >>>>>>> path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" >>>>>>> dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 >>>>>>> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>>>> type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 >>>>>>> success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 >>>>>>> a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 >>>>>>> gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) >>>>>>> ses=4294967295 comm="postgres" >>>>>>> exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" >>>>>>> subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>>>> >>>>>>> whereas in [3] - the build just before the selinux package update, >>>>>>> these errors did not occur. >>>>>>> >>>>>>> Looks like alongside enabling selinux a policy update is required. >>>>>>> >>>>>>> thanks >>>>>>> >>>>>>> >>>>>>> [0] >>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ >>>>>>> [1] >>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/ >>>>>>> [2] >>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/audit/audit.log/*view*/ >>>>>>> [3] >>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/ >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <[email protected]> wrote: >>>>>>> >>>>>>>> I think this is a regression causing rh-postgress to fail to start >>>>>>>> on selinux conf. >>>>>>>> the issue is probably with the selinux packages >>>>>>>> >>>>>>>> I ran lago locally to debug and ssh-ed to the vms and this is the >>>>>>>> output from the processes start: >>>>>>>> >>>>>>>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >>>>>>>> postfix/postdrop[9028]: warning: unable to look up public/pickup: No >>>>>>>> such >>>>>>>> file or directory >>>>>>>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >>>>>>>> postfix/postdrop[9029]: warning: unable to look up public/pickup: No >>>>>>>> such >>>>>>>> file or directory >>>>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>>>> polkitd[2720]: Registered Authentication Agent for >>>>>>>> unix-process:9033:93610 >>>>>>>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 >>>>>>>> --fallback], ob >>>>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Starting PostgreSQL database server... >>>>>>>> -- Subject: Unit rh-postgresql95-postgresql.service has begun >>>>>>>> start-up >>>>>>>> -- Defined-By: systemd >>>>>>>> -- Support: >>>>>>>> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >>>>>>>> -- >>>>>>>> -- Unit rh-postgresql95-postgresql.service has begun starting up. >>>>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>>>> postgresql-ctl[9041]: postgres cannot access the server configuration >>>>>>>> file >>>>>>>> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": >>>>>>>> Permission d >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> postgresql-ctl[9041]: pg_ctl: could not start server >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> postgresql-ctl[9041]: Examine the log output. >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: rh-postgresql95-postgresql.service: control process exited, >>>>>>>> code=exited status=1 >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Failed to start PostgreSQL database server. >>>>>>>> -- Subject: Unit rh-postgresql95-postgresql.service has failed >>>>>>>> -- Defined-By: systemd >>>>>>>> -- Support: >>>>>>>> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >>>>>>>> -- >>>>>>>> -- Unit rh-postgresql95-postgresql.service has failed. >>>>>>>> -- >>>>>>>> -- The result is failed. >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Unit rh-postgresql95-postgresql.service entered failed >>>>>>>> state. >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: rh-postgresql95-postgresql.service failed. >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> polkitd[2720]: Unregistered Authentication Agent for >>>>>>>> unix-process:9033:93610 (system bus name :1.160, object path >>>>>>>> /org/freedesktop/PolicyKit1/Authent >>>>>>>> Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Started Session 51 of user root. >>>>>>>> -- Subject: Unit session-51.scope has finished start-up >>>>>>>> -- Defined-By: systemd >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Secure log: >>>>>>>> >>>>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>>>> polkitd[2720]: Registered Authentication Agent for >>>>>>>> unix-process:9033:93610 >>>>>>>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], >>>>>>>> object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale >>>>>>>> en_US.UTF-8) >>>>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>>>> polkitd[2720]: Unregistered Authentication Agent for >>>>>>>> unix-process:9033:93610 (system bus name :1.160, object path >>>>>>>> /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) >>>>>>>> (disconnected from bus) >>>>>>>> >>>>>>>> after setenforce: >>>>>>>> >>>>>>>> root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 >>>>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >>>>>>>> start rh-postgresql95-postgresql.service >>>>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >>>>>>>> status rh-postgresql95-postgresql.service >>>>>>>> ● rh-postgresql95-postgresql.service - PostgreSQL database server >>>>>>>> Loaded: loaded >>>>>>>> (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; >>>>>>>> vendor preset: disabled) >>>>>>>> Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s >>>>>>>> ago >>>>>>>> Process: 9137 >>>>>>>> ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl >>>>>>>> start -D >>>>>>>> ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) >>>>>>>> Process: 9134 >>>>>>>> ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir >>>>>>>> %N (code=exited, status=0/SUCCESS) >>>>>>>> Main PID: 9143 (postgres) >>>>>>>> CGroup: /system.slice/rh-postgresql95-postgresql.service >>>>>>>> ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D >>>>>>>> /var/opt/rh/rh-postgresql95/lib/pgsql/data >>>>>>>> ├─9144 postgres: logger process >>>>>>>> ├─9146 postgres: checkpointer process >>>>>>>> ├─9147 postgres: writer process >>>>>>>> ├─9148 postgres: wal writer process >>>>>>>> ├─9149 postgres: autovacuum launcher process >>>>>>>> └─9150 postgres: stats collector process >>>>>>>> >>>>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Starting PostgreSQL database server... >>>>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>>>> postgresql-ctl[9137]: LOG: redirecting log output to logging collector >>>>>>>> process >>>>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>>>> postgresql-ctl[9137]: HINT: Future log output will appear in directory >>>>>>>> "pg_log". >>>>>>>> Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine >>>>>>>> systemd[1]: Started PostgreSQL database server. >>>>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>>>> >>>>>>>> Not sure who deals with this configuration but this is a blocker as >>>>>>>> upgrade from release is failing for both ovirt-engine and vdsm. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Thanks Greg >>>>>>>>> >>>>>>>>> I will check this >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Is there any way you can run >>>>>>>>>> "systemctl status rh-postgresql95-postgresql.service" and >>>>>>>>>> "journalctl -xe" >>>>>>>>>> like it suggests? >>>>>>>>>> The logs below don't give any indication why it failed to start, >>>>>>>>>> afaict. >>>>>>>>>> >>>>>>>>>> On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi >>>>>>>>>>> >>>>>>>>>>> I receive this error message both in CQ and check_patch: >>>>>>>>>>> >>>>>>>>>>> 2019-02-16 16:28:06,874-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.services.systemd systemd.state:130 starting >>>>>>>>>>> service rh-postgresql95-postgresql >>>>>>>>>>> 2019-02-16 16:28:06,874-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: >>>>>>>>>>> ('/usr/bin/systemctl', 'start', >>>>>>>>>>> 'rh-postgresql95-postgresql.service'), executable='None', >>>>>>>>>>> cwd='None', env=None >>>>>>>>>>> 2019-02-16 16:28:07,913-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.services.systemd plugin.executeRaw:863 >>>>>>>>>>> execute-result: ('/usr/bin/systemctl', 'start', >>>>>>>>>>> 'rh-postgresql95-postgresql.service'), rc=1 >>>>>>>>>>> 2019-02-16 16:28:07,914-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.services.systemd plugin.execute:921 >>>>>>>>>>> execute-output: ('/usr/bin/systemctl', 'start', >>>>>>>>>>> 'rh-postgresql95-postgresql.service') stdout: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2019-02-16 16:28:07,914-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.services.systemd plugin.execute:926 >>>>>>>>>>> execute-output: ('/usr/bin/systemctl', 'start', >>>>>>>>>>> 'rh-postgresql95-postgresql.service') stderr: >>>>>>>>>>> Job for rh-postgresql95-postgresql.service failed because the >>>>>>>>>>> control process exited with error code. See "systemctl status >>>>>>>>>>> rh-postgresql95-postgresql.service" and "journalctl -xe" for >>>>>>>>>>> details. >>>>>>>>>>> >>>>>>>>>>> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction >>>>>>>>>>> transaction.abort:119 aborting 'File transaction for >>>>>>>>>>> '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' >>>>>>>>>>> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context >>>>>>>>>>> context._executeMethod:143 method exception >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line >>>>>>>>>>> 133, in _executeMethod >>>>>>>>>>> method['method']() >>>>>>>>>>> File >>>>>>>>>>> "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", >>>>>>>>>>> line 201, in _misc >>>>>>>>>>> self._provisioning.provision() >>>>>>>>>>> File >>>>>>>>>>> "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", >>>>>>>>>>> line 498, in provision >>>>>>>>>>> self.restartPG() >>>>>>>>>>> File >>>>>>>>>>> "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", >>>>>>>>>>> line 399, in restartPG >>>>>>>>>>> state=state, >>>>>>>>>>> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line >>>>>>>>>>> 141, in state >>>>>>>>>>> service=name, >>>>>>>>>>> RuntimeError: Failed to start service 'rh-postgresql95-postgresql' >>>>>>>>>>> 2019-02-16 16:28:07,918-0500 ERROR otopi.context >>>>>>>>>>> context._executeMethod:152 Failed to execute stage 'Misc >>>>>>>>>>> configuration': Failed to start service 'rh-postgresql95-postgresql' >>>>>>>>>>> 2019-02-16 16:28:07,958-0500 DEBUG >>>>>>>>>>> otopi.plugins.otopi.debug.debug_failure.debug_failure >>>>>>>>>>> debug_failure._notification:100 tcp connections: >>>>>>>>>>> id uid local foreign state pid exe >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> What can cause it? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Galit >>>>>>>>>>> >>>>>>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change-queue-tester/12916/testReport/junit/(root)/001_initialize_engine/running_tests___upgrade_from_release_suite_el7_x86_64___test_initialize_engine/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_standard-check-patch/detail/ovirt-system-tests_standard-check-patch/3207/pipeline/101 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Galit >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> GALIT ROSENTHAL >>>>>>>>>>> >>>>>>>>>>> SOFTWARE ENGINEER >>>>>>>>>>> >>>>>>>>>>> Red Hat >>>>>>>>>>> >>>>>>>>>>> <https://www.redhat.com/> >>>>>>>>>>> >>>>>>>>>>> [email protected] T: 972-9-7692230 >>>>>>>>>>> <https://red.ht/sig> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: >>>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/QNDG65M6UPEXTCT3HXORRTZ67RVXH653/ >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> GREG SHEREMETA >>>>>>>>>> >>>>>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>>>>>> >>>>>>>>>> Red Hat NA >>>>>>>>>> >>>>>>>>>> <https://www.redhat.com/> >>>>>>>>>> >>>>>>>>>> [email protected] IRC: gshereme >>>>>>>>>> <https://red.ht/sig> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> GALIT ROSENTHAL >>>>>>>>> >>>>>>>>> SOFTWARE ENGINEER >>>>>>>>> >>>>>>>>> Red Hat >>>>>>>>> >>>>>>>>> <https://www.redhat.com/> >>>>>>>>> >>>>>>>>> [email protected] T: 972-9-7692230 >>>>>>>>> <https://red.ht/sig> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/YROV4PLNBTOWKYMT2EL25CN3C26HOU2R/ >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- [email protected] >>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/CSNQENF4J6ZQJGS5T4QQMRRBDGZG6J4L/ >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Associate Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>>> >>> > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o. > -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> [email protected] <https://red.ht/sig>
_______________________________________________ Devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/FHCVVLQHCLOVX747NH5L4GCT6W4GQTPO/
