[ovirt-users] Re: oVirt Node 4.4.1.1 Cockpit Hyperconverged Gluster deploy fails insufficient free space no matter how small the volume is set
Dear Strahil, Dominique and Edward: I reimaged the three hosts with ovirt-node-ng-installer-4.4.1-2020071311.el8.iso just to be sure everything was stock (I had upgraded from v4.4) and attempted a redeploy with all suggested changes EXCEPT replacing "package" with "dnf" --> same failure. I then made Strahil's recommended replacement of "package" with "dnf" and the Gluster deployment succeeded through that section of main.yml only to fail a little later at: - name: Install python-yaml package for Debian systems package: name: python-yaml state: present when: ansible_distribution == "Debian" or ansible_distribution == "Ubuntu" I found this notable given that I had not replaced "package" with "dnf" in the prior section: - name: Change to Install lvm tools for debian systems. package: name: thin-provisioning-tools state: present when: ansible_distribution == "Debian" or ansible_distribution == "Ubuntu" and deployment had not failed here. Anyhow, I deleted the two Debian statements as I am deploying from Node (CentOS based), cleaned up, cleaned up my drives ('dmsetup remove eui.xxx...' and 'wipefs --all --force /dev/nvme0n1 /dev/nvmeXn1 ...') and redeployed again. This time Gluster deployment seemed to execute main.yml OK only to fail in a new file, vdo_create.yml: TASK [gluster.infra/roles/backend_setup : Install VDO dependencies] task path: /etc/ansible/roles/gluster.infra/roles/backend_setup/tasks/vdo_create.yml:26 fatal: [fmov1n1.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} fatal: [fmov1n3.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} fatal: [fmov1n2.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} Expecting that this might continue, I have been looking into the documentation of how "package" works and if I can find a root cause for this rather than reviewing n *.yml files and replacing "package" with "dnf" in all of them. Thank you VERY much to Strahil for helping me! If Strahil or anyone else has any additional troubleshooting tips, suggestions, insight or solutions I am all ears. I will continue to update as I progress. Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3JTZX2OF4JTGRECMZLZXZQT5IWR4PFSG/
[ovirt-users] Re: oVirt Node 4.4.1.1 Cockpit Hyperconverged Gluster deploy fails insufficient free space no matter how small the volume is set
Thank you Strahil. I think I edited the oVirt Node Cockpit Hyperconverged Wizard Gluster Deployment Ansible playbook as detailed in your post and received the following new failure: TASK [gluster.infra/roles/backend_setup : Change to Install lvm tools for RHEL systems.] *** fatal: [fmov1n3.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} fatal: [fmov1n1.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} fatal: [fmov1n2.sn.dtcorp.com]: FAILED! => {"changed": false, "msg": "The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."} Any further assistance is most appreciated!!! Respectfully, Charles --- Gluster Deployment Ansible Playbook hc_nodes: hosts: fmov1n1.sn.dtcorp.com: gluster_infra_volume_groups: - vgname: gluster_vg_nvme0n1 pvname: /dev/mapper/vdo_nvme0n1 - vgname: gluster_vg_nvme2n1 pvname: /dev/mapper/vdo_nvme2n1 - vgname: gluster_vg_nvme1n1 pvname: /dev/mapper/vdo_nvme1n1 gluster_infra_mount_devices: - path: /gluster_bricks/engine lvname: gluster_lv_engine vgname: gluster_vg_nvme0n1 - path: /gluster_bricks/data lvname: gluster_lv_data vgname: gluster_vg_nvme2n1 - path: /gluster_bricks/vmstore lvname: gluster_lv_vmstore vgname: gluster_vg_nvme1n1 gluster_infra_vdo: - name: vdo_nvme0n1 device: /dev/nvme0n1 slabsize: 2G logicalsize: 1000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M - name: vdo_nvme2n1 device: /dev/nvme2n1 slabsize: 32G logicalsize: 5000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M - name: vdo_nvme1n1 device: /dev/nvme1n1 slabsize: 32G logicalsize: 5000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M blacklist_mpath_devices: - nvme0n1 - nvme2n1 - nvme1n1 gluster_infra_thick_lvs: - vgname: gluster_vg_nvme0n1 lvname: gluster_lv_engine size: '100%PVS' gluster_infra_thinpools: - vgname: gluster_vg_nvme2n1 thinpoolname: gluster_thinpool_gluster_vg_nvme2n1 poolmetadatasize: 3G - vgname: gluster_vg_nvme1n1 thinpoolname: gluster_thinpool_gluster_vg_nvme1n1 poolmetadatasize: 3G gluster_infra_lv_logicalvols: - vgname: gluster_vg_nvme2n1 thinpool: gluster_thinpool_gluster_vg_nvme2n1 lvname: gluster_lv_data lvsize: '100%PVS' - vgname: gluster_vg_nvme1n1 thinpool: gluster_thinpool_gluster_vg_nvme1n1 lvname: gluster_lv_vmstore lvsize: '100%PVS' fmov1n2.sn.dtcorp.com: gluster_infra_volume_groups: - vgname: gluster_vg_nvme0n1 pvname: /dev/mapper/vdo_nvme0n1 - vgname: gluster_vg_nvme2n1 pvname: /dev/mapper/vdo_nvme2n1 - vgname: gluster_vg_nvme1n1 pvname: /dev/mapper/vdo_nvme1n1 gluster_infra_mount_devices: - path: /gluster_bricks/engine lvname: gluster_lv_engine vgname: gluster_vg_nvme0n1 - path: /gluster_bricks/data lvname: gluster_lv_data vgname: gluster_vg_nvme2n1 - path: /gluster_bricks/vmstore lvname: gluster_lv_vmstore vgname: gluster_vg_nvme1n1 gluster_infra_vdo: - name: vdo_nvme0n1 device: /dev/nvme0n1 slabsize: 2G logicalsize: 1000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M - name: vdo_nvme2n1 device: /dev/nvme2n1 slabsize: 32G logicalsize: 5000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M - name: vdo_nvme1n1 device: /dev/nvme1n1 slabsize: 32G logicalsize: 5000G blockmapcachesize: 128M emulate512: 'off' writepolicy: auto maxDiscardSize: 16M blacklist_mpath_devices: - nvme0n1 - nvme2n1 - nvme1n1 gluster_infra_thick_lvs: - vgname: gluster_vg_nvme0n1 lvname: gluster_lv_engine size: '100%PVS' gluster_infra_thinpools: - vgname: gluster_vg_nvme2n1 thinpoolname: gluster_thinpool_gluster_vg_nvme2n1 poolmetadatasize: 3G
[ovirt-users] Re: oVirt Node 4.4.1.1 Cockpit Hyperconverged Gluster deploy fails insufficient free space no matter how small the volume is set
Output of pvdisplay for each of three hosts below. Node 1: --- Physical volume --- PV Name /dev/mapper/vdo_nvme2n1 VG Name gluster_vg_nvme2n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID tKg74P-klP8-o2sX-XCER-wcHf-XW9Q-mFViNT --- Physical volume --- PV Name /dev/mapper/vdo_nvme1n1 VG Name gluster_vg_nvme1n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID wXyN5p-LaX3-9b9f-3RbH-j1B6-sXfT-UZ0BG7 --- Physical volume --- PV Name /dev/mapper/vdo_nvme0n1 VG Name gluster_vg_nvme0n1 PV Size 100.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 25599 Free PE 25599 Allocated PE 0 PV UUID gTHFgm-NU5J-LJWJ-DyIb-ecm7-85Cq-OedKeX --- Physical volume --- PV Name /dev/mapper/luks-3890d311-7c61-43ae-98a5-42c0318e735f VG Name onn PV Size <221.92 GiB / not usable 0 Allocatable yes PE Size 4.00 MiB Total PE 56811 Free PE 10897 Allocated PE 45914 PV UUID FqWsAT-hxAO-UCgq-PA7e-m0W1-3Jrw-XGnLf1 --- Node 2: --- Physical volume --- PV Name /dev/mapper/vdo_nvme2n1 VG Name gluster_vg_nvme2n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID KR4c82-465u-B22g-2Q95-4l81-1urD-iqvBRt --- Physical volume --- PV Name /dev/mapper/vdo_nvme1n1 VG Name gluster_vg_nvme1n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID sEABVg-tCRU-zW8n-pfPW-p5aj-XbBt-IjsTp1 --- Physical volume --- PV Name /dev/mapper/vdo_nvme0n1 VG Name gluster_vg_nvme0n1 PV Size 100.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 25599 Free PE 25599 Allocated PE 0 PV UUID NLRTl5-05ol-6zcH-ZjAS-T82n-hcow-20LYEL --- Physical volume --- PV Name /dev/mapper/luks-7d42e806-af06-4a72-96b7-de77f76e562f VG Name onn PV Size <221.92 GiB / not usable 0 Allocatable yes PE Size 4.00 MiB Total PE 56811 Free PE 10897 Allocated PE 45914 PV UUID O07nNl-yd7X-Gh8x-2d4b-lRME-bz21-OjCykI --- Node 3: --- Physical volume --- PV Name /dev/mapper/vdo_nvme2n1 VG Name gluster_vg_nvme2n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID 4Yji7W-LuIv-Y2Aq-oD8t-wBwO-VaXY-9coNN0 --- Physical volume --- PV Name /dev/mapper/vdo_nvme1n1 VG Name gluster_vg_nvme1n1 PV Size 1000.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 255999 Free PE 255999 Allocated PE 0 PV UUID rTEqJ0-SkWm-Ge05-iz97-ZOoT-AdYY-L6uHtN --- Physical volume --- PV Name /dev/mapper/vdo_nvme0n1 VG Name gluster_vg_nvme0n1 PV Size 100.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 25599 Free PE 25599 Allocated PE 0 PV UUID AoJ9h9-vNYG-IgXQ-gSdB-aYWi-Nzl0-JPiQU3 --- Physical volume --- PV Name /dev/mapper/luks-5ac3e150-55c1-4fc2-acd4-f2861c3d2e0a VG Name onn PV Size <221.92 GiB / not usable 0 Allocatable yes PE Size 4.00 MiB Total PE 56811 Free PE 10897 Allocated PE 45914 PV UUID N3HLbG-kUIb-5I98-UfZX-eG9A-qnHi-J4tWWi --- My apologies for the delay (I am UTC-4). Thanks so very much for your input Ritesh! Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to use
[ovirt-users] oVirt Node 4.4.1.1 Cockpit Hyperconverged Gluster deploy fails insufficient free space no matter how small the volume is set
Hi, Deploying oVirt 4.4.1.1 via Cockpit --> Hosted Engine --> Hyperconverged fails at Gluster deployment: TASK [gluster.infra/roles/backend_setup : Create thick logical volume] * failed: [fmov1n3.sn.dtcorp.com] (item={'vgname': 'gluster_vg_nvme0n1', 'lvname': 'gluster_lv_engine', 'size': '100G'}) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "err": " Volume group \"gluster_vg_nvme0n1\" has insufficient free space (25599 extents): 25600 required.\n", "index": 0, "item": {"lvname": "gluster_lv_engine", "size": "100G", "vgname": "gluster_vg_nvme0n1"}, "msg": "Creating logical volume 'gluster_lv_engine' failed", "rc": 5} failed: [fmov1n1.sn.dtcorp.com] (item={'vgname': 'gluster_vg_nvme0n1', 'lvname': 'gluster_lv_engine', 'size': '100G'}) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "err": " Volume group \"gluster_vg_nvme0n1\" has insufficient free space (25599 extents): 25600 required.\n", "index": 0, "item": {"lvname": "gluster_lv_engine", "size": "100G", "vgname": "gluster_vg_nvme0n1"}, "msg": "Creating logical volume 'gluster_lv_engine' failed", "rc": 5} failed: [fmov1n2.sn.dtcorp.com] (item={'vgname': 'gluster_vg_nvme0n1', 'lvname': 'gluster_lv_engine', 'size': '100G'}) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "err": " Volume group \"gluster_vg_nvme0n1\" has insufficient free space (25599 extents): 25600 required.\n", "index": 0, "item": {"lvname": "gluster_lv_engine", "size": "100G", "vgname": "gluster_vg_nvme0n1"}, "msg": "Creating logical volume 'gluster_lv_engine' failed", "rc": 5} Deployment is on 3 count Dell PowerEdge R740xd with 5 count 1.6TB NVMe drives per host. Deployment is only to three as JBOD, 1 drive per node per volume (engine, data, vmstore) utilizing VDO. Thus, deploying even a 100G volume to 1.6TB drive fails with "insufficient free space" error. I suspect this might have to do with the Ansible playbook deploying Gluster mishandling the logical volume creation due to the rounding error as described here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/nofreeext If I can provide any additional information, logs, etc. please ask. Also, if anyone has experience/suggestions with Gluster config for hyperconverged setup on NVMe drives I would greatly appreciate any pearls of wisdom. Thank you so very much for any assistance! Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3AARZD4VBNNHWNWRCVD2QNWQZJYY5AL5/
[ovirt-users] Re: Deploy Hosted Engine fails at "Set VLAN ID at datacenter level"
Thank you Guillaume! My mistake. Resolved. I am now having further issues I believe to be because I am using tagged VLANs with NIC teaming. It appears that teaming is not well supported in oVirt - is that accurate, and that I should rebuild using bonds. If you have any experience to shed light on this it is much appreciated. [ INFO ] TASK [ovirt.hosted_engine_setup : Fail with error description] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host fmov1n1.bcn.dtcorp.com installation failed. Failed to configure management network on the host., code 9000: Failed to verify Power Management configuration for Host fmov1n1.bcn.dtcorp.com., fix accordingly and re-deploy."} Thanks so very much, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LPUANZALZ7L5ZC26XE65AKFJCPUINU2N/
[ovirt-users] Re: Deploy Hosted Engine fails at "Set VLAN ID at datacenter level"
Hello, I am having this same issue and have inserted the three new lines from https://github.com/ansible/ansible/issues/66858 into "/lib/python2.7/site-packages/ansible/modules/cloud/ovirt/ovirt_network.py" yet the issue still occurs when attempting deployement. I have rebooted the oVirt Node hosts since patching to no avail. Do I need to insert the fix elsewhere or take other action to properly apply this fix? Thank you very much for your assistance. Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKWVUWZBISU75RYCUCNC3DMBE3IX5NRJ/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
Dear Andrej and Martin: I just wanted to follow-up and thank you both so very much. I was able to update to the latest 4.1.9 and that resolved my issues - engine started just fine, all vm's are up. My apologies for the delayed update, I had back surgery and was out a while. Again, I really appreciate everyone's responses - most helpful!!! Sincerely, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KIPPBJRCVUBYRI65ILP7PHBHOIVR7ZYZ/
[ovirt-users] oVirt Node 4.1 at www.ovirt.org/node/ is 404
I am trying to download Node 4.1 from https://www.ovirt.org/node but am receiving a 404 error. Is there an alternative link to try for the latest stable release? Thanks, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHR755UBG3O5TMGZES2PWSQZ4BOS6XGH/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
Thank you again so very much Andrej! I am going to try this right now... Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RM45ON3RADJD5WPM3IHGQ3EF6PWOQ4PX/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
OK, I think the inability to change /var/run/ovirt-hosted-engine-ha/vm.conf might have something to do with HE configuration having been moved to shared storage? (https://ovirt.org/develop/release-management/features/sla/hosted-engine-configuration-on-shared-storage/) I am using NFS shares on a Synology cluster Any ideas how to fix memSize=4 and successfully boot HE? Thanks yet again, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/V3KKIKWANCOZWPMXQAAZVZTOUBDGXM5J/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
I cannot get /var/run/ovirt-hosted-engine-ha/vm.conf to keep my edit of "memSize=4" to "memSize=4096" --> it keeps reverting. I have confirmed that HE is down --> suggestions? Thanks, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7ZEQMREJ56QRTC4Y4DY7ZWAS7ICU6FWX/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
Andrej, thank you so much!!! I knew I was having issues with X11, and was trying to troubleshoot that here. Your suggestion to setup port forwarding, with some tweaks, worked!!! Thank you thank you. I am now able to see that HostedEngine is not booting, it states "cannot allocate kernel buffer", which I was able to find referenced in this forum post https://lists.ovirt.org/pipermail/users/2017-December/085631.html Apparently /var/run/ovirt-hosted-engine-ha/vm.conf got changed to boot HostedEngine with only 4MB of RAM, not 4GB. I am now trying to get HostedEngine to boot with 4GB. I am sure I will need some more assistance before I am through here! Thanks again and I will repost with update. Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JEO7MUCA6445TCXPVAH2CRFVT5N5FAGM/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
> Hi, > > You could try accessing the engine VM using the VNC. > First, set the VNC password using 'hosted-engine --add-console-password', > and then connect to the host using a VNC viewer, for example: > 'remove-viewer vnc://HOST_IP:5900' > > The liveliness check just checks if the web UI is running and reachable. > If the VNC works, try checking if the 'ovirt-engine' service is running on > the VM and that the network is configured properly. > > > Andrej > > On 30 May 2018 at 18:31, https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5W4N6BGVCCABUQTFZFOAE6ITYVCD2E2H/
[ovirt-users] Re: Non-responsive vm's due to crashed host and hosted vm liveliness check fails
Hi, I am still working to resolve my issue - is there any further detail or clarification I can provide that might help? I really appreciate your time. Thank you, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XE37L37BGTWIWXOYP5PTYM64I2NNKRO5/
[ovirt-users] Non-responsive vm's due to crashed host and hosted vm liveliness check fails
Dear Community: The local drive on a host running ovirt-node-ng-4.1.9 in a three node cluster failed. I have production JIRA and Postres running on it at the time, not in HA, just simple vm's. Storage is via NFS on a Synology NAS. Hosted Engine was on a different host, but JIRA and Postgres vm's showed nonresponsive. I tried different things but then stupidly thought that upgrading my Hosted Engine would reinitialize the vm's that were on the failed host. Update of HE seemed to go well (output below) but now my Hosted Engine, while up, fails the liveliness check and the web management console is unavailable. I cannot console into the HE from the host it is running on. Below are the results of my attempts to console into the Hosted Engine. Please help! I have search forums, lists and Google but have not been able to fix this. My coworkers and manager are anxious. --- When I try "hosted-engine --console" after setting the console password I get The engine VM is running on this host Connected to domain HostedEngine Escape character is ^] _ The prompt is non-responsive except for the escape character key combo. --- "virsh -r list" gives ID 3, Name: HostedEngine, State: running "virsh -r console HostedEngine" gives Connected to domain HostedEngine Escape character is ^] error: operation forbidden: read only access prevents virDomainOpenConsole "virsh -r vncdisplay HostedEngine" gives "0:0" and returns me to prompt --- I am SSHed into the host running my Hosted Engine from a CentOS7 minimal install with packages xorg-x11-server-Xorg, xorg-x11-xauth and xorg-x11-apps installed. The result of "grep -i X11Forwarding /etc/ssh/sshd_config" shows it set to "Yes". I SSH into the host using "ssh -Y r...@xxx.xxx.xxx.xxx" I am logged into the CentOS7 minimal install as root. I know root is poor practice but was trying to minimize anything that could be causing an issue. --- Below are the results of my attempt to update Hosted Engine (slightly redacted to remove personal info): --== CONFIGURATION PREVIEW ==-- Default SAN wipe after delete : False Firewall manager: firewalld Update Firewall : True Host FQDN : ovengineint.xdomainx.tld Upgrade packages: True Engine database secured connection : False Engine database user name : engine Engine database name: engine Engine database host: localhost Engine database port: 5432 Engine database host name validation: False Engine installation : True PKI organization: xdomainx.tld Set up ovirt-provider-ovn : True Configure WebSocket Proxy : True DWH installation: True DWH database secured connection : False DWH database host : localhost DWH database user name : ovirt_engine_history DWH database name : ovirt_engine_history DWH database port : 5432 DWH database host name validation : False Configure Image I/O Proxy : True Configure VMConsole Proxy : True --== SUMMARY ==-- [ INFO ] Restarting httpd Web access is enabled at: http://ovengineint.xdomainx.tld:80/ovirt-engine https://ovengineint.xdomainx.tld:443/ovirt-engine Internal CA XX:XX:XX:XX... SSH fingerprint: SHA256:xx... --== END OF SUMMARY ==-- [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180502165652-88pkpi.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180502170149-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully Again, thank you so very much for any suggestions! I have found many answers on this mailing list archive to be of great insight and help. Respectfully, Charles ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VOISG2XLUQ6SZ2OGEIKAXNDN6CKLLCRB/