Just update for others I and Kaustav were doing more investigation and found an issue which was introduced by https://gerrit.ovirt.org/#/c/107683/, because of removing over escaped parameters. Right now working on the fix.
On Fri, Mar 27, 2020 at 4:54 PM Kaustav Majumder <[email protected]> wrote: > Unfortunately no artifact stdout files point to the playbook being run. > There are others like when adding a new host and checking for host updates > but none pertaining to create-brick . The latest one is this pointing to (i > guess) host updates checker > > [root@localhost artifacts] # less > 83029f4c-702d-11ea-b308-68ecc5af4344/stdout > PLAY [all] > ********************************************************************* > > TASK [Gathering Facts] > ********************************************************* > ok: [10.70.35.88] > > TASK [ovirt-host-check-upgrade : Check for system updates (remove empty > lines)] *** > this message. > changed: [10.70.35.88] > > TASK [ovirt-host-check-upgrade : Remove 'Obsoleting' title from file] > ********** > ok: [10.70.35.88] > > TASK [ovirt-host-check-upgrade : Get yum updates file] > ************************* > ok: [10.70.35.88] > > TASK [ovirt-host-check-upgrade : Get yum updates content] > ********************** > ok: [10.70.35.88] > > TASK [ovirt-host-check-upgrade : Delete yum_updates file from host] > ************ > changed: [10.70.35.88] > > PLAY RECAP > ********************************************************************* > 10.70.35.88 : ok=6 changed=2 unreachable=0 > failed=0 skipped=0 rescued=0 ignored=0 > > On Fri, Mar 27, 2020 at 9:07 PM Martin Necas <[email protected]> wrote: > >> The journal looks okay to me... >> Maybe the playbook failed please go >> to ovirt-engine/share/ovirt-engine/ansible-runner-service-project/artifacts/ >> and find the newest folder and check stdout file. >> >> On Fri, Mar 27, 2020 at 4:16 PM Martin Necas <[email protected]> wrote: >> >>> Sorry did not notice that link to the journal. >>> >>> On Fri, Mar 27, 2020 at 4:13 PM Martin Necas <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> my bet is on the issue with ansible-runner please try to install new >>>> runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and >>>> restart service (`systemctl restart ansible-runner-service.service`). >>>> if it won't help please use `journalctl -u >>>> ansible-runner-service.service -f` to gather info from the runner. >>>> The log which is mentioned in engine.log is not created in dev env >>>> everything is logged to journal. >>>> >>>> Martin Necas >>>> >>>> On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <[email protected]> >>>>> wrote: >>>>> >>>>>> HI Martin, >>>>>> Below error after rebasing you patch >>>>>> 2020-03-27 18:34:24,996+05 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) >>>>>> [] >>>>>> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >>>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to >>>>>> execute >>>>>> call to start playbook. >>>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >>>>>> brick role. Please check logs for more details: >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>>> 2020-03-27 18:34:25,772+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>> EngineException: Failed to execute Ansible create brick role: 'Failed to >>>>>> execute call to start playbook. '. Please check logs for more details: >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>>> (Failed with error GeneralException and code 100) >>>>>> 2020-03-27 18:34:25,852+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for >>>>>> command >>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >>>>>> 2020-03-27 18:34:25,863+05 ERROR >>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on >>>>>> host >>>>>> 10.70.35.17 of cluster Default. >>>>>> >>>>> >>>>> Dana/Martin, any idea about the issue? >>>>> >>>>>> >>>>>> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Let me know if you require something else >>>>>>>> journalctl -u ansible-runner-service -> >>>>>>>> https://pastebin.com/e6tnrzSB >>>>>>>> engine.log - > https://pastebin.com/bVv7FxLB >>>>>>>> >>>>>>> >>>>>>> Hmm, could you please try to reproduce with below patch, maybe there >>>>>>> will be more information in stderr: >>>>>>> >>>>>>> https://gerrit.ovirt.org/108012 >>>>>>> >>>>>>> Thanks, >>>>>>> Martin >>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Could you please share journal and engine.log? We need to >>>>>>>>> correlate and find out which error in journal is correlated to this >>>>>>>>> issue? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Martin >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Artur, >>>>>>>>>> I changed the files mentioned in the mail >>>>>>>>>> >>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/MXPFVUAVE6UYOYRHNLKIUQG54DPNPJ2D/ >>>>>>>>>> >>>>>>>>>> but unfortunately it is not the same error as described in the >>>>>>>>>> mail and changing it gives me the same error. >>>>>>>>>> >>>>>>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Kaustav, >>>>>>>>>>> You might be hitting a known issue. >>>>>>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>>>>>> title: '[ovirt-devel] workaround for host deploy failure >>>>>>>>>>> (get_events)' >>>>>>>>>>> Artur >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>>>>>> ``` >>>>>>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: >>>>>>>>>>>> VDSAction group >>>>>>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 >>>>>>>>>>>> ERROR >>>>>>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] >>>>>>>>>>>> (default >>>>>>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed >>>>>>>>>>>> to >>>>>>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>> task-29) >>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible >>>>>>>>>>>> create >>>>>>>>>>>> brick role. Please check logs for more details: >>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>> task-29) >>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>>>>>> EngineException: Failed to execute Ansible create brick role. >>>>>>>>>>>> Please check >>>>>>>>>>>> logs for more details: >>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>>>>>> 15:43:07,526+05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>> task-29) >>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for >>>>>>>>>>>> command >>>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>>>>>> 15:43:07,530+05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 >>>>>>>>>>>> on host >>>>>>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>>>>>> *The corresponding log is not created* >>>>>>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>>>>>> No such file or directory >>>>>>>>>>>> *checking ansible-runner status gives* >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s >>>>>>>>>>>> 130 ± systemctl status ansible-runner-service.service ● >>>>>>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: >>>>>>>>>>>> loaded >>>>>>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; >>>>>>>>>>>> vendor >>>>>>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 >>>>>>>>>>>> 15:42:56 >>>>>>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: >>>>>>>>>>>> 4915) >>>>>>>>>>>> Memory: 62.1M CGroup: >>>>>>>>>>>> /system.slice/ansible-runner-service.service >>>>>>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b >>>>>>>>>>>> localhost:50001 -w 2 >>>>>>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 >>>>>>>>>>>> localhost.localdomain >>>>>>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 >>>>>>>>>>>> 15:42:58 >>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>> content-type >>>>>>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>> content-type >>>>>>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>> content-type >>>>>>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>> content-type >>>>>>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 >>>>>>>>>>>> localhost.localdomain >>>>>>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST >>>>>>>>>>>> /api/v1/playbooks/create-brick.yml* >>>>>>>>>>>> >>>>>>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Kaustav Majumder >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>> List Archives: >>>>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/P2GWVT2HBSTFAHZ55ZV3TA7YALWBNNTA/ >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Artur Socha >>>>>>>>>>> >>>>>>>>>>> Senior Software Engineer, RHV >>>>>>>>>>> >>>>>>>>>>> Red Hat <https://www.redhat.com> >>>>>>>>>>> <https://www.redhat.com> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Kaustav Majumder >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/ULE6X7ZF6NMB5MK3QEZEU5BQ2MBQVESP/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Martin Perina >>>>>>>>> Manager, Software Engineering >>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Kaustav Majumder >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Martin Perina >>>>>>> Manager, Software Engineering >>>>>>> Red Hat Czech s.r.o. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kaustav Majumder >>>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> > > -- > > Thanks, > > Kaustav Majumder >
_______________________________________________ Devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/X7RNMTVBATJGVDMSD7WI7Q7443GYZXVK/
