Hi, my bet is on the issue with ansible-runner please try to install new runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and restart service (`systemctl restart ansible-runner-service.service`). if it won't help please use `journalctl -u ansible-runner-service.service -f` to gather info from the runner. The log which is mentioned in engine.log is not created in dev env everything is logged to journal.
Martin Necas On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <[email protected]> wrote: > > > On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <[email protected]> > wrote: > >> HI Martin, >> Below error after rebasing you patch >> 2020-03-27 18:34:24,996+05 INFO >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [] >> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >> 2020-03-27 18:34:25,771+05 ERROR >> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to execute >> call to start playbook. >> 2020-03-27 18:34:25,771+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >> brick role. Please check logs for more details: >> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >> 2020-03-27 18:34:25,772+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >> EngineException: Failed to execute Ansible create brick role: 'Failed to >> execute call to start playbook. '. Please check logs for more details: >> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >> (Failed with error GeneralException and code 100) >> 2020-03-27 18:34:25,852+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for command >> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >> 2020-03-27 18:34:25,863+05 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >> 10.70.35.17 of cluster Default. >> > > Dana/Martin, any idea about the issue? > >> >> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <[email protected]> wrote: >> >>> >>> >>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder <[email protected]> >>> wrote: >>> >>>> Hi, >>>> Let me know if you require something else >>>> journalctl -u ansible-runner-service -> https://pastebin.com/e6tnrzSB >>>> engine.log - > https://pastebin.com/bVv7FxLB >>>> >>> >>> Hmm, could you please try to reproduce with below patch, maybe there >>> will be more information in stderr: >>> >>> https://gerrit.ovirt.org/108012 >>> >>> Thanks, >>> Martin >>> >>>> >>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <[email protected]> >>>> wrote: >>>> >>>>> Could you please share journal and engine.log? We need to correlate >>>>> and find out which error in journal is correlated to this issue? >>>>> >>>>> Thanks, >>>>> Martin >>>>> >>>>> >>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Artur, >>>>>> I changed the files mentioned in the mail >>>>>> >>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/MXPFVUAVE6UYOYRHNLKIUQG54DPNPJ2D/ >>>>>> >>>>>> but unfortunately it is not the same error as described in the mail >>>>>> and changing it gives me the same error. >>>>>> >>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Kaustav, >>>>>>> You might be hitting a known issue. >>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>> title: '[ovirt-devel] workaround for host deploy failure (get_events)' >>>>>>> Artur >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>> ``` >>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction >>>>>>>> group >>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR >>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to >>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>> task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create >>>>>>>> brick role. Please check logs for more details: >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>> task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>> EngineException: Failed to execute Ansible create brick role. Please >>>>>>>> check >>>>>>>> logs for more details: >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>> 15:43:07,526+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>> task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for >>>>>>>> command >>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>> 15:43:07,530+05 ERROR >>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on >>>>>>>> host >>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>> *The corresponding log is not created* >>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>> No such file or directory >>>>>>>> *checking ansible-runner status gives* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s 130 ± >>>>>>>> systemctl status ansible-runner-service.service ● >>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: >>>>>>>> loaded >>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; >>>>>>>> vendor >>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 >>>>>>>> 15:42:56 >>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915) >>>>>>>> Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service >>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 >>>>>>>> -w 2 >>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain >>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58 >>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: >>>>>>>> 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: >>>>>>>> 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: >>>>>>>> 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain >>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml* >>>>>>>> >>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Kaustav Majumder >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- [email protected] >>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/P2GWVT2HBSTFAHZ55ZV3TA7YALWBNNTA/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Artur Socha >>>>>>> >>>>>>> Senior Software Engineer, RHV >>>>>>> >>>>>>> Red Hat <https://www.redhat.com> >>>>>>> <https://www.redhat.com> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kaustav Majumder >>>>>> _______________________________________________ >>>>>> Devel mailing list -- [email protected] >>>>>> To unsubscribe send an email to [email protected] >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/ULE6X7ZF6NMB5MK3QEZEU5BQ2MBQVESP/ >>>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> >>>> Kaustav Majumder >>>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >> >> >> -- >> >> Thanks, >> >> Kaustav Majumder >> > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. >
_______________________________________________ Devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/ER7MV5ZIAKWQYJY4CJUOMYCIUUHT5C5F/
