Hi, Thanks to Martin Necas we have fixed the issue in CreateBrickCommand. Posted patch below. Let's get this reviewed and merged soon. https://gerrit.ovirt.org/#/c/108021/
On Sat, Mar 28, 2020 at 12:23 AM Martin Necas <[email protected]> wrote: > Just update for others I and Kaustav were doing more investigation and > found an issue which was introduced by > https://gerrit.ovirt.org/#/c/107683/, because of removing over escaped > parameters. > Right now working on the fix. > > On Fri, Mar 27, 2020 at 4:54 PM Kaustav Majumder <[email protected]> > wrote: > >> Unfortunately no artifact stdout files point to the playbook being run. >> There are others like when adding a new host and checking for host updates >> but none pertaining to create-brick . The latest one is this pointing to (i >> guess) host updates checker >> >> [root@localhost artifacts] # less >> 83029f4c-702d-11ea-b308-68ecc5af4344/stdout >> PLAY [all] >> ********************************************************************* >> >> TASK [Gathering Facts] >> ********************************************************* >> ok: [10.70.35.88] >> >> TASK [ovirt-host-check-upgrade : Check for system updates (remove empty >> lines)] *** >> this message. >> changed: [10.70.35.88] >> >> TASK [ovirt-host-check-upgrade : Remove 'Obsoleting' title from file] >> ********** >> ok: [10.70.35.88] >> >> TASK [ovirt-host-check-upgrade : Get yum updates file] >> ************************* >> ok: [10.70.35.88] >> >> TASK [ovirt-host-check-upgrade : Get yum updates content] >> ********************** >> ok: [10.70.35.88] >> >> TASK [ovirt-host-check-upgrade : Delete yum_updates file from host] >> ************ >> changed: [10.70.35.88] >> >> PLAY RECAP >> ********************************************************************* >> 10.70.35.88 : ok=6 changed=2 unreachable=0 >> failed=0 skipped=0 rescued=0 ignored=0 >> >> On Fri, Mar 27, 2020 at 9:07 PM Martin Necas <[email protected]> wrote: >> >>> The journal looks okay to me... >>> Maybe the playbook failed please go >>> to ovirt-engine/share/ovirt-engine/ansible-runner-service-project/artifacts/ >>> and find the newest folder and check stdout file. >>> >>> On Fri, Mar 27, 2020 at 4:16 PM Martin Necas <[email protected]> wrote: >>> >>>> Sorry did not notice that link to the journal. >>>> >>>> On Fri, Mar 27, 2020 at 4:13 PM Martin Necas <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> my bet is on the issue with ansible-runner please try to install new >>>>> runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and >>>>> restart service (`systemctl restart ansible-runner-service.service`). >>>>> if it won't help please use `journalctl -u >>>>> ansible-runner-service.service -f` to gather info from the runner. >>>>> The log which is mentioned in engine.log is not created in dev env >>>>> everything is logged to journal. >>>>> >>>>> Martin Necas >>>>> >>>>> On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> HI Martin, >>>>>>> Below error after rebasing you patch >>>>>>> 2020-03-27 18:34:24,996+05 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) >>>>>>> [] >>>>>>> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >>>>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>>> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to >>>>>>> execute >>>>>>> call to start playbook. >>>>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >>>>>>> brick role. Please check logs for more details: >>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>>>> 2020-03-27 18:34:25,772+05 ERROR >>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>> EngineException: Failed to execute Ansible create brick role: 'Failed to >>>>>>> execute call to start playbook. '. Please check logs for more details: >>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>>>> (Failed with error GeneralException and code 100) >>>>>>> 2020-03-27 18:34:25,852+05 ERROR >>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for >>>>>>> command >>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >>>>>>> 2020-03-27 18:34:25,863+05 ERROR >>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on >>>>>>> host >>>>>>> 10.70.35.17 of cluster Default. >>>>>>> >>>>>> >>>>>> Dana/Martin, any idea about the issue? >>>>>> >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Let me know if you require something else >>>>>>>>> journalctl -u ansible-runner-service -> >>>>>>>>> https://pastebin.com/e6tnrzSB >>>>>>>>> engine.log - > https://pastebin.com/bVv7FxLB >>>>>>>>> >>>>>>>> >>>>>>>> Hmm, could you please try to reproduce with below patch, maybe >>>>>>>> there will be more information in stderr: >>>>>>>> >>>>>>>> https://gerrit.ovirt.org/108012 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Martin >>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Could you please share journal and engine.log? We need to >>>>>>>>>> correlate and find out which error in journal is correlated to this >>>>>>>>>> issue? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Artur, >>>>>>>>>>> I changed the files mentioned in the mail >>>>>>>>>>> >>>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/MXPFVUAVE6UYOYRHNLKIUQG54DPNPJ2D/ >>>>>>>>>>> >>>>>>>>>>> but unfortunately it is not the same error as described in the >>>>>>>>>>> mail and changing it gives me the same error. >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Kaustav, >>>>>>>>>>>> You might be hitting a known issue. >>>>>>>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>>>>>>> title: '[ovirt-devel] workaround for host deploy failure >>>>>>>>>>>> (get_events)' >>>>>>>>>>>> Artur >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>>>>>>> ``` >>>>>>>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: >>>>>>>>>>>>> VDSAction group >>>>>>>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 >>>>>>>>>>>>> ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] >>>>>>>>>>>>> (default >>>>>>>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed >>>>>>>>>>>>> to >>>>>>>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>>> task-29) >>>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible >>>>>>>>>>>>> create >>>>>>>>>>>>> brick role. Please check logs for more details: >>>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>>> task-29) >>>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>>>>>>> EngineException: Failed to execute Ansible create brick role. >>>>>>>>>>>>> Please check >>>>>>>>>>>>> logs for more details: >>>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>>>>>>> 15:43:07,526+05 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default >>>>>>>>>>>>> task-29) >>>>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back >>>>>>>>>>>>> for command >>>>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>>>>>>> 15:43:07,530+05 ERROR >>>>>>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick >>>>>>>>>>>>> brick-1 on host >>>>>>>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>>>>>>> *The corresponding log is not created* >>>>>>>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± >>>>>>>>>>>>> less >>>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>>>>>>> No such file or directory >>>>>>>>>>>>> *checking ansible-runner status gives* >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s >>>>>>>>>>>>> 130 ± systemctl status ansible-runner-service.service ● >>>>>>>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: >>>>>>>>>>>>> loaded >>>>>>>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; >>>>>>>>>>>>> disabled; vendor >>>>>>>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 >>>>>>>>>>>>> 15:42:56 >>>>>>>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: >>>>>>>>>>>>> 4915) >>>>>>>>>>>>> Memory: 62.1M CGroup: >>>>>>>>>>>>> /system.slice/ansible-runner-service.service >>>>>>>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b >>>>>>>>>>>>> localhost:50001 -w 2 >>>>>>>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 >>>>>>>>>>>>> localhost.localdomain >>>>>>>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 >>>>>>>>>>>>> 15:42:58 >>>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>>> content-type >>>>>>>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>>> content-type >>>>>>>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>>> content-type >>>>>>>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: >>>>>>>>>>>>> 127.0.0.1 - >>>>>>>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, >>>>>>>>>>>>> content-type >>>>>>>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 >>>>>>>>>>>>> localhost.localdomain >>>>>>>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST >>>>>>>>>>>>> /api/v1/playbooks/create-brick.yml* >>>>>>>>>>>>> >>>>>>>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Kaustav Majumder >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>> List Archives: >>>>>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/P2GWVT2HBSTFAHZ55ZV3TA7YALWBNNTA/ >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Artur Socha >>>>>>>>>>>> >>>>>>>>>>>> Senior Software Engineer, RHV >>>>>>>>>>>> >>>>>>>>>>>> Red Hat <https://www.redhat.com> >>>>>>>>>>>> <https://www.redhat.com> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Kaustav Majumder >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list -- [email protected] >>>>>>>>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: >>>>>>>>>>> https://lists.ovirt.org/archives/list/[email protected]/message/ULE6X7ZF6NMB5MK3QEZEU5BQ2MBQVESP/ >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Martin Perina >>>>>>>>>> Manager, Software Engineering >>>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Kaustav Majumder >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Martin Perina >>>>>>>> Manager, Software Engineering >>>>>>>> Red Hat Czech s.r.o. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Kaustav Majumder >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >> >> -- >> >> Thanks, >> >> Kaustav Majumder >> > -- Thanks, Kaustav Majumder
_______________________________________________ Devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/N32F4WIF7KMG2VJPJLKTE4RF6AI37HC5/
