[ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Sorry about the late response. I looked at the logs. These errors are originating from posix-acl translator - *[2019-11-17 07:55:47.090065] E [MSGID: 115050] [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496: LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6 (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6), client: CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, error-xlator: data_fast-access-control [Permission denied][2019-11-17 07:55:47.090174] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:36,gid:36,perm:1,ngrps:3), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050] [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497: LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7 (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7), client: CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, error-xlator: data_fast-access-control [Permission denied][2019-11-17 07:55:47.090299] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:36,gid:36,perm:1,ngrps:3), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission denied]* Jiffin/Raghavendra Talur, Can you help? -Krutika On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov wrote: > Hi Nir,All, > > it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached > screenshot of oVirt running on v7 gluster). > It seems strange that both my serious issues with oVirt are related to > gluster issue (1st gluster v3 to v5 migration and now this one). > > I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all > nodes. > Now both Engine and all my VMs are back online - so if you hit issues with > 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before > deciding to wipe everything. > > @Krutika, > > I guess you will ask for the logs, so let's switch to gluster-users about > this one ? > > Best Regards, > Strahil Nikolov > > В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov < > hunter86...@yahoo.com> написа: > > > Hi Krutika, > > I have enabled TRACE log level for the volume data_fast, > > but the issue is not much clear: > FUSE reports: > > [2019-11-25 21:31:53.478130] I [MSGID: 133022] > [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of > gfid=6d9ed2e5-d4f2-4749-839b-2f1 > 3a68ed472 from backend > [2019-11-25 21:32:43.564694] W [MSGID: 114031] > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: > remote operation failed. Path: > /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 > (----) [Permission denied] > [2019-11-25 21:32:43.565653] W [MSGID: 114031] > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: > remote operation failed. Path: > /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 > (----) [Permission denied] > [2019-11-25 21:32:43.565689] W [MSGID: 114031] > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: > remote operation failed. Path: > /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 > (----) [Permission denied] > [2019-11-25 21:32:43.565770] E [MSGID: 133010] > [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on > shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae > [Permission denied] > [2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk] > 0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae > fd=0x7fbf40005ea8 (Permission denied) > > > While the BRICK logs on ovirt1/gluster1 report: > 2019-11-25 21:32:43.564177] D [MSGID: 0] [io-threads.c:376:iot_schedule] > 0-data_fast-io-threads: LOOKUP scheduled as fast priority fop > [2019-11-25 21:32:43.564194] T [MSGID: 0] > [defaults.c:2008:default_lookup_resume] 0-stack-trace: stack-address: > 0x7fc02c00bbf8, winding from data_fast-io-threads to data_fast-upcall > [2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup] > 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall > to data_fast-leases > [2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup] > 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases > to
[ovirt-users] Re: Overt Networking VM not pinging
So , I assume the hosts can communicate with the Engine , A & PTR records are resolved on host level. Correct me if I'm wrong. Have you run tcpdump on the ovirtmgmt bridge , hunting for icmp requests from your clients ? Are you using firewalld or iptables as a firewall ? Best Regards,Strahil Nikolov В неделя, 1 декември 2019 г., 22:09:09 ч. Гринуич+2, Vijay Sachdeva написа: Thanks for replying, I have single host with 4 interfaces. Two of the interfaces are bonded and connected to bridge “ovirtmgmt” and other bridge(VLAN based). None of the networks works neither “ovirtmgmt” untagged, and YES I checked they are not out of sync. Although Ovirt-engine, using VDSM setup that but no luck. Any suggestion, would be a great help..!! Thanks Vijay Sachdeva From: Strahil Nikolov Date: Monday, 2 December 2019 at 1:27 AM To: , Vijay Sachdeva Subject: Re: [ovirt-users] Re: Overt Networking VM not pinging Have you checked in each host's -> Network Interfaces -> NIC if ovirtmgmt is out of sync ? Best Regards, Strahil Nikolov В неделя, 1 декември 2019 г., 20:29:06 ч. Гринуич+2, Vijay Sachdeva написа: Hi Team, Anyone having any Solution for this, please revert. Thanks On Sun, 1 Dec 2019 at 12:29 AM, Vijay Sachdeva wrote: Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WHBSXZASHUMFDTQMZC4AHX62BCWY5SD5/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XDIE5GEOZT6TXYHGF36BMMWG7GBB26U/
[ovirt-users] Re: Overt Networking VM not pinging
Thanks for replying, I have single host with 4 interfaces. Two of the interfaces are bonded and connected to bridge “ovirtmgmt” and other bridge(VLAN based). None of the networks works neither “ovirtmgmt” untagged, and YES I checked they are not out of sync. Although Ovirt-engine, using VDSM setup that but no luck. Any suggestion, would be a great help..!! Thanks Vijay Sachdeva From: Strahil Nikolov Date: Monday, 2 December 2019 at 1:27 AM To: , Vijay Sachdeva Subject: Re: [ovirt-users] Re: Overt Networking VM not pinging Have you checked in each host's -> Network Interfaces -> NIC if ovirtmgmt is out of sync ? Best Regards, Strahil Nikolov В неделя, 1 декември 2019 г., 20:29:06 ч. Гринуич+2, Vijay Sachdeva написа: Hi Team, Anyone having any Solution for this, please revert. Thanks On Sun, 1 Dec 2019 at 12:29 AM, Vijay Sachdeva wrote: Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WHBSXZASHUMFDTQMZC4AHX62BCWY5SD5/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KOAQFQ32DL2RXSZTWYF3YDEDZBPETKXC/
[ovirt-users] Re: Overt Networking VM not pinging
Have you checked in each host's -> Network Interfaces -> NIC if ovirtmgmt is out of sync ? Best Regards,Strahil Nikolov В неделя, 1 декември 2019 г., 20:29:06 ч. Гринуич+2, Vijay Sachdeva написа: Hi Team, Anyone having any Solution for this, please revert. Thanks On Sun, 1 Dec 2019 at 12:29 AM, Vijay Sachdeva wrote: Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WHBSXZASHUMFDTQMZC4AHX62BCWY5SD5/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/666KSN7C235H3LZZAIEXRH4LGUNLP3WF/
[ovirt-users] Re: NFS Storage Domain on OpenMediaVault
Does sanlock user has rights on the ./dom_md/ids ? Check the sanlock.service for issues.journalctl -u sanlock.service Best Regards,Strahil Nikolov В неделя, 1 декември 2019 г., 17:22:21 ч. Гринуич+2, rw...@ropeguru.com написа: I have a clean install with openmediavault as backend NFS and cannot get it to work. Keep getting permission errors even though I created a vdsm user and kvm group; and they are the owners of the directory on OMV with full permissions. The directory gets created on the NFS side for the host, but then get the permission error and is removed form the host but the directory structure is left on the NFS server. Logs: >From the engine: Error while executing action New NFS Storage Domain: Unexpected exception >From the oVirt node log: 2019-11-29 10:03:02 136998 [30025]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/192.168.1.56:_export_Datastore-oVirt/f38b19e4-8060-4467-860b-09cf606ccc15/dom_md/ids 2019-11-29 10:03:02 136998 [30025]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. File system on Openmediavault: drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 . drwxr-xr-x 9 root root 4096 Nov 27 20:56 .. drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 f38b19e4-8060-4467-860b-09cf606ccc15 drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 . drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 .. drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 dom_md drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 images drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 . drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 .. -rw-rw+ 1 vdsm kvm 0 Nov 29 10:03 ids -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 inbox -rw-rw+ 1 vdsm kvm 0 Nov 29 10:03 leases -rw-rw-r--+ 1 vdsm kvm 343 Nov 29 10:03 metadata -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 outbox -rw-rw+ 1 vdsm kvm 1302528 Nov 29 10:03 xleases ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ILKNT57F6VUHEVKMOACLLQRAO3J364MC/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IULXFX3UCJ3CEFHLXL5XJ7YAX64N3EDP/
[ovirt-users] Re: 3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
I think that you can go on with the installation (as far as I remember , next phase is the HostedEngine deployment) on the same node. You should not use the single node setup , but the other one.At the end - the engine (once migrated to gluster volume and started up by the ovirt-ha-broker/ovirt-ha-agent) will detect the gluster cluster, once you add all nodes in oVirt. Then you won't have any issues to manage the storage (although I prefer the cli approach). Best Regards,Strahil Nikolov В неделя, 1 декември 2019 г., 16:37:23 ч. Гринуич+2, tho...@hoberg.net написа: Three Gen8 HP360 recalled from retirement with single 1TB TLC SATA SSD for boot and oVirt /engine and 7x4TB HDD RAID6 for /vmstore and /data, 10Gbit NICs and network. All CentOS 7.7 updated daily. These machines may not be used exclusively for oVirt so I don't want to re-install the OS, when an oVirt setup fails: Instead I try my best to clean up the nodes when doing another oVirt installation run. They ran oVirt for a week or two using a completely distinct set of storage, so they are fundamentally sound, but we wanted higher storage capacity so I swapped everything and re-installed CentOS very much the same way as before. The first oVirt setup went smoothly but the cluster crumbled without much usage. I won't go into details here, because I didn't want to investigate for now, instead I focussed on redoing the installation and cleaning up the old setup. I know the docs actually recommend starting with wiped hardware, but operationally that would be a show-stopper for the intended use case. So I cleaned up the best I can (ovirt-hosted-engine-cleanup with and without redoing the whole Gluster storage setup, where apart from SSD caching not working, I don't have issues). Undoing the network changes in such a way that the oVirt HCI wizard ceases complaining is a bit more involved. I typically run: - vdsm-tool ovn-unconfigure - vdsm-tool clear-nets (now need to switch to the console) - vdsm-tool remove-config and then I still need to edit /etc/sysconfig/network-scripts/ifcfg- to bring the physical adapter back to life. Sometimes I still need to remove the ovirtmgmnt bridge manually etc. Whether I remove and redo the Gluster as a bit of an effect in re-installation, but it doesn't make a difference in what follows. So here is where I am currently getting stuck consistently: The wizard is gone through preparing the Gluster storage (which is completely functional at that point), has created the local VM on the installation node, installed the Postgres database, filled it etc. basically has oVirt up and running with the primary Gluster node and now would like to add the second and third nodes. At that point I get "Connection lost" in the Web-Wizard, evidently as a consequence of Ansible fiddling around heavily to set up the local bridge for the VM. I remember that for the scripted variant of the setup it is recommended to run the script behind 'screen' or 'tmux' in order to ensure its execution isn't interrupted by that. But for the GUI variant, evidently there *should* be some other type of potection, perhaps via the re-connecting nature of HTTP... Pushing the "Reconnect" button on the GUI at that point doesn't return you to the point of the setup, but only offers to redeploy, while the HostedEngineLocal is still there and running. I ssh'd into the machine and started looking for errors and warning and saw that the installation had gone rather far without incidence. OTOPI had completely finished the WildFly server is up and running the Postgres database fully installed and running smoothly, the only thing I can find is that it's trying to add the additional gluster nodes, but complains that these nodes (quotes gluster-UUIDs) are not part of the "cluster". An investigation into the Postgres database shows, that the 'gluster_server' table indeed only has the primary node in it. I don't know what part of the process should have added the other two nodes, but there seems to be no *remaining* connectivity issue with the Gluster members. I installed gscli and connected to all three nodes and volumes without issue. I am guessing at this point, that the complex rewiring of the software defined network is causing a temporary issue and a race condition that I don't know how to recover from. Since the oVirt management GUI is actually fully operational and can be reached from the primary node via the temporary bridge, I went into the GUI and even managed to add the additional two nodes without any problems. Their installation went through without any issues, they showed up in the gluster_servers table on Postgress and basically the installation could have proceeded from that point, except... that I don't know how to restart the process from that point: It still has to 'beam' the local VM into the Gluster storage and restart it there. I have gone through the process three times now,
[ovirt-users] Re: hyperconverged single node with SSD cache fails gluster creation
When I first deployed my oVirt lab (v4.2.7 was latest and greatest) the ansible playbook didn't work for me.So I decided to stop the gluster processes on one of the nodes, Wipe all LVM and recreate it manually. Finally , I have managed to use my SSD for write-back cache - but I found out that if your Chunk size is larger than the default limit - it will never push it to the spinning disks. For details you can check 1668163 – LVM cache cannot flush buffer,change cache type or lvremove LV (CachePolicy 'cleaner' also doesn't work) As we use either 'replica 2 arbiter 1' (old name replica 3 arbiter 1) or a pure replica 3 , we can afford a gluster node go 'pouf' as long as we have decent bandwidth and we use sharding. So far I have changed my brick layout at least twice (for the cluster) without the VMs being affected - so you can still try to do the caching, but please check the comments in #1668163 about the chunk size of the cache. Best Regards,Strahil Nikolov В неделя, 1 декември 2019 г., 16:02:36 ч. Гринуич+2, Thomas Hoberg написа: Hi Gobinda, unfortunately it's long gone, because I went back to an un-cached setup. It was mostly a trial anyway, I had to re-do the 3-node HCI because it had died rather horribly on me (a repeating issue I have so far had on distinct sets of hardware, that I am still trying to hunt down... separate topic). And since it was a blank(ed) set of servers, I just decided to try the SSD cache, to see if the Ansible script generation issue had been sorted out as described from upstream. I was rather encouraged to see that the Ansible script now had these changes included, that URS had described as becoming necessary with a new Ansible version. It doesn't actually make a lot of sense in the setup, because the SSD cache is a single Samsung EVO 860 1TB unit while the storage is a RAID6 out of 7 4TB 2.5" drives (per server): Both have similar bandwidth, IOPS would be very much workload dependent (the 2nd SSD I intended to use as a mirror was unfortunately cut from the budget). It has space left over because the OS doesn't need that much, but I don't dare use a single SSD as a write-back cache, especially because the RAID controller (HP420i) hides all wear information and doesn't seem to pass TRIM either and for write-through I'm not sure it would do noticeably better than the RAID controller (I configured that not to cache the SSD, too). So after it failed, I simply went back to no-cache for now. This HCI cluster is using relatively low-power hardware recalled from retirement that will host functional VMs, not high-performance workloads. They are well equipped with RAM and that's always the fastest cache anyway. I guess you should be able to add and remove the SSD as cache layer at any time during the operation, because it's at a level oVirt doesn't manage and I'd love to see examples as to how it's done. Especially the removal part would be important to know, if your SSD signals unexpected levels of wear and you need to swap them out on the fly. If I hit across another opportunity to test (most likely a single node), I will update here and make sure to collect a full set of log files including the ansible main config file. Thank you for your interest and the follow-up, Thomas ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y45BAH7PXJN6C6HXG4VDX4TRRPCH6TOX/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/J3A5ROG3BS4S6H7S7GXOTUYUZMIUSX6P/
[ovirt-users] Re: Overt Networking VM not pinging
Hi Team, Anyone having any Solution for this, please revert. Thanks On Sun, 1 Dec 2019 at 12:29 AM, Vijay Sachdeva wrote: > Dear Community, > > > > I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got > successfully added to engine and setup host network also done. When trying > to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even > able to ping it’s host or any other machine on that same network. Also > added a VLAN network which is passed via same uplink of Node interface > where “Ovirtmgmt” is passed, that is also not working. > > > > > > Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would > this be a problem? > > > > Any help would be highly appreciated. > > > > Thanks > > > > Vijay Sachdeva > > Senior Manager – Service Delivery > > > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WHBSXZASHUMFDTQMZC4AHX62BCWY5SD5/
[ovirt-users] Re: NFS Storage Domain on OpenMediaVault
On Sun, Dec 1, 2019 at 5:22 PM wrote: > I have a clean install with openmediavault as backend NFS and cannot get > it to work. Keep getting permission errors even though I created a vdsm > user and kvm group; and they are the owners of the directory on OMV with > full permissions. > > The directory gets created on the NFS side for the host, but then get the > permission error and is removed form the host but the directory structure > is left on the NFS server. > > Logs: > > From the engine: > > Error while executing action New NFS Storage Domain: Unexpected exception > > From the oVirt node log: > > 2019-11-29 10:03:02 136998 [30025]: open error -13 EACCES: no permission > to open /rhev/data-center/mnt/192.168.1.56: > _export_Datastore-oVirt/f38b19e4-8060-4467-860b-09cf606ccc15/dom_md/ids > 2019-11-29 10:03:02 136998 [30025]: check that daemon user sanlock 179 > group sanlock 179 has access to disk or file. > Make sure sanlock user is a member of vdsm and kvm groups id -a sanlock should list also kvm and vdsm. This is something that vdsm-tool configure --force systemctl restart libvirtd systemctl restart vdsm should set if not set already. > > File system on Openmediavault: > > drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 . > drwxr-xr-x 9 root root 4096 Nov 27 20:56 .. > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 > f38b19e4-8060-4467-860b-09cf606ccc15 > > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 . > drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 .. > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 dom_md > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 images > > drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 . > drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 .. > -rw-rw+ 1 vdsm kvm0 Nov 29 10:03 ids > -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 inbox > -rw-rw+ 1 vdsm kvm0 Nov 29 10:03 leases > -rw-rw-r--+ 1 vdsm kvm 343 Nov 29 10:03 metadata > -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 outbox > -rw-rw+ 1 vdsm kvm 1302528 Nov 29 10:03 xleases > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ILKNT57F6VUHEVKMOACLLQRAO3J364MC/ > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VJOP2HRCE63QKCELUW2KUUH3PFSNRKF5/
[ovirt-users] NFS Storage Domain on OpenMediaVault
I have a clean install with openmediavault as backend NFS and cannot get it to work. Keep getting permission errors even though I created a vdsm user and kvm group; and they are the owners of the directory on OMV with full permissions. The directory gets created on the NFS side for the host, but then get the permission error and is removed form the host but the directory structure is left on the NFS server. Logs: From the engine: Error while executing action New NFS Storage Domain: Unexpected exception From the oVirt node log: 2019-11-29 10:03:02 136998 [30025]: open error -13 EACCES: no permission to open /rhev/data-center/mnt/192.168.1.56:_export_Datastore-oVirt/f38b19e4-8060-4467-860b-09cf606ccc15/dom_md/ids 2019-11-29 10:03:02 136998 [30025]: check that daemon user sanlock 179 group sanlock 179 has access to disk or file. File system on Openmediavault: drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 . drwxr-xr-x 9 root root 4096 Nov 27 20:56 .. drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 f38b19e4-8060-4467-860b-09cf606ccc15 drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 . drwxrwsrwx+ 3 vdsm kvm 4096 Nov 29 10:03 .. drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 dom_md drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 images drwxrwsr-x+ 2 vdsm kvm 4096 Nov 29 10:03 . drwxrwsr-x+ 4 vdsm kvm 4096 Nov 29 10:03 .. -rw-rw+ 1 vdsm kvm0 Nov 29 10:03 ids -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 inbox -rw-rw+ 1 vdsm kvm0 Nov 29 10:03 leases -rw-rw-r--+ 1 vdsm kvm 343 Nov 29 10:03 metadata -rw-rw+ 1 vdsm kvm 16777216 Nov 29 10:03 outbox -rw-rw+ 1 vdsm kvm 1302528 Nov 29 10:03 xleases ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ILKNT57F6VUHEVKMOACLLQRAO3J364MC/
[ovirt-users] 3node HCI fails when HostedEngineLocal is trying to add additional Gluster members
Three Gen8 HP360 recalled from retirement with single 1TB TLC SATA SSD for boot and oVirt /engine and 7x4TB HDD RAID6 for /vmstore and /data, 10Gbit NICs and network. All CentOS 7.7 updated daily. These machines may not be used exclusively for oVirt so I don't want to re-install the OS, when an oVirt setup fails: Instead I try my best to clean up the nodes when doing another oVirt installation run. They ran oVirt for a week or two using a completely distinct set of storage, so they are fundamentally sound, but we wanted higher storage capacity so I swapped everything and re-installed CentOS very much the same way as before. The first oVirt setup went smoothly but the cluster crumbled without much usage. I won't go into details here, because I didn't want to investigate for now, instead I focussed on redoing the installation and cleaning up the old setup. I know the docs actually recommend starting with wiped hardware, but operationally that would be a show-stopper for the intended use case. So I cleaned up the best I can (ovirt-hosted-engine-cleanup with and without redoing the whole Gluster storage setup, where apart from SSD caching not working, I don't have issues). Undoing the network changes in such a way that the oVirt HCI wizard ceases complaining is a bit more involved. I typically run: - vdsm-tool ovn-unconfigure - vdsm-tool clear-nets (now need to switch to the console) - vdsm-tool remove-config and then I still need to edit /etc/sysconfig/network-scripts/ifcfg- to bring the physical adapter back to life. Sometimes I still need to remove the ovirtmgmnt bridge manually etc. Whether I remove and redo the Gluster as a bit of an effect in re-installation, but it doesn't make a difference in what follows. So here is where I am currently getting stuck consistently: The wizard is gone through preparing the Gluster storage (which is completely functional at that point), has created the local VM on the installation node, installed the Postgres database, filled it etc. basically has oVirt up and running with the primary Gluster node and now would like to add the second and third nodes. At that point I get "Connection lost" in the Web-Wizard, evidently as a consequence of Ansible fiddling around heavily to set up the local bridge for the VM. I remember that for the scripted variant of the setup it is recommended to run the script behind 'screen' or 'tmux' in order to ensure its execution isn't interrupted by that. But for the GUI variant, evidently there *should* be some other type of potection, perhaps via the re-connecting nature of HTTP... Pushing the "Reconnect" button on the GUI at that point doesn't return you to the point of the setup, but only offers to redeploy, while the HostedEngineLocal is still there and running. I ssh'd into the machine and started looking for errors and warning and saw that the installation had gone rather far without incidence. OTOPI had completely finished the WildFly server is up and running the Postgres database fully installed and running smoothly, the only thing I can find is that it's trying to add the additional gluster nodes, but complains that these nodes (quotes gluster-UUIDs) are not part of the "cluster". An investigation into the Postgres database shows, that the 'gluster_server' table indeed only has the primary node in it. I don't know what part of the process should have added the other two nodes, but there seems to be no *remaining* connectivity issue with the Gluster members. I installed gscli and connected to all three nodes and volumes without issue. I am guessing at this point, that the complex rewiring of the software defined network is causing a temporary issue and a race condition that I don't know how to recover from. Since the oVirt management GUI is actually fully operational and can be reached from the primary node via the temporary bridge, I went into the GUI and even managed to add the additional two nodes without any problems. Their installation went through without any issues, they showed up in the gluster_servers table on Postgress and basically the installation could have proceeded from that point, except... that I don't know how to restart the process from that point: It still has to 'beam' the local VM into the Gluster storage and restart it there. I have gone through the process three times now, with absolutely identical results. I could use some help how to recover from that situation, which looks like a race condition, nothing a re-installation of everything would really resolve. In the mean-time, I'll try the scripted variant on 'screen' to see if that fares better. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
[ovirt-users] Re: hyperconverged single node with SSD cache fails gluster creation
Hi Gobinda, unfortunately it's long gone, because I went back to an un-cached setup. It was mostly a trial anyway, I had to re-do the 3-node HCI because it had died rather horribly on me (a repeating issue I have so far had on distinct sets of hardware, that I am still trying to hunt down... separate topic). And since it was a blank(ed) set of servers, I just decided to try the SSD cache, to see if the Ansible script generation issue had been sorted out as described from upstream. I was rather encouraged to see that the Ansible script now had these changes included, that URS had described as becoming necessary with a new Ansible version. It doesn't actually make a lot of sense in the setup, because the SSD cache is a single Samsung EVO 860 1TB unit while the storage is a RAID6 out of 7 4TB 2.5" drives (per server): Both have similar bandwidth, IOPS would be very much workload dependent (the 2nd SSD I intended to use as a mirror was unfortunately cut from the budget). It has space left over because the OS doesn't need that much, but I don't dare use a single SSD as a write-back cache, especially because the RAID controller (HP420i) hides all wear information and doesn't seem to pass TRIM either and for write-through I'm not sure it would do noticeably better than the RAID controller (I configured that not to cache the SSD, too). So after it failed, I simply went back to no-cache for now. This HCI cluster is using relatively low-power hardware recalled from retirement that will host functional VMs, not high-performance workloads. They are well equipped with RAM and that's always the fastest cache anyway. I guess you should be able to add and remove the SSD as cache layer at any time during the operation, because it's at a level oVirt doesn't manage and I'd love to see examples as to how it's done. Especially the removal part would be important to know, if your SSD signals unexpected levels of wear and you need to swap them out on the fly. If I hit across another opportunity to test (most likely a single node), I will update here and make sure to collect a full set of log files including the ansible main config file. Thank you for your interest and the follow-up, Thomas <>___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y45BAH7PXJN6C6HXG4VDX4TRRPCH6TOX/
[ovirt-users] Overt Networking VM not pinging
Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery *IndiQus Technologies * ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RYAA2NJSCCSU2AD7SGEDTMBOPND25B3T/
[ovirt-users] Re: Upgrade ovirt from 3.4 to 4.3
Hello Luigi, You can upgrade to the latest minor of 3.4 and then upgrade step by step to each major (3.5 3.6 4.0 4.1 4.2 4.3) It's a long procedure, requires several downtimes for upgrading the cluster compatibility level, but shouldn't be so hard. Luca Il dom 1 dic 2019, 12:05 ha scritto: > Good morning, > > i have a difficult enviroment with 20 Hypervisors based on ovirt 3.4.3-1 > and i would like to reach the 4.3 version. Which are the best steps to > achieve these objective? > > Thanks in advance > > Luigi > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/LUA3Q7QGAEJJRSY7UGSMSKJ77CVPMESW/ > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WADZIQEMIVOXVH7FV34CVNAUJF72OAZV/
[ovirt-users] Overt Networking VM not pinging
Dear Community, I have installed Ovirt Engine 4.3 and Ovirt Node 4.3. Node got successfully added to engine and setup host network also done. When trying to ping host using “ovirtmgmt” as vNic profile for a VM, it is not even able to ping it’s host or any other machine on that same network. Also added a VLAN network which is passed via same uplink of Node interface where “Ovirtmgmt” is passed, that is also not working. Although this vnet is vnic type is VirtIO and state shows “UNKNOWN”, would this be a problem? Any help would be highly appreciated. Thanks Vijay Sachdeva Senior Manager – Service Delivery IndiQus Technologies O +91 11 4055 1411 | M +91 8826699409 www.indiqus.com ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U5TSYZSKND2BXIWNSEOG4WVVXZSROZBG/
[ovirt-users] Upgrade ovirt from 3.4 to 4.3
Good morning, i have a difficult enviroment with 20 Hypervisors based on ovirt 3.4.3-1 and i would like to reach the 4.3 version. Which are the best steps to achieve these objective? Thanks in advance Luigi ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LUA3Q7QGAEJJRSY7UGSMSKJ77CVPMESW/
[ovirt-users] Re: Moving HostedEngine
The installation will fail if you attempt to install HE into an existing storage domain (or worse, it will corrupt the domain). As per the docs you will need a dedicated domain of at least 74GB for hosted_storage. https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_command_line/preparing_storage_for_rhv_she_cli_deploy On Wed, 27 Nov 2019 23:05:14 + Joseph Goldman wrote So I can't host OTHER VM's on this gluster volume? If its already a running GLuser for other VMs i can't now re-reploy HE in that gluster volume? On 2019-11-28 3:27 AM, Alan G wrote: I've had to do this a couple of times and always ended up with a working system in the end. As a fall back option (although I've never had to use it) I have a backup engine VM running completely outside of oVIrt (ESXi host in my case). Then if the hosted_engine deploy fails for any reason you can restore onto the backup vm as a temp solution while you work through the hosted engine deploy issues. A few things that come to mind: - * You will need a dedicated gluster volume for hosted_storage and it needs to be replica+arbiter. * Make sure you put the cluster in global maint mode before performing the engine backup, I recall having issues with the restore when I didn't do that. * Migrate all other VMs off the host running Engine before doing the backup. This will be the host you will restore onto. On Wed, 27 Nov 2019 09:46:23 + Joseph Goldman mailto:jos...@goldman.id.au wrote Hi List, In one of my installs, I set up the first storage domain (and where the HostedEngine is) on a bigger NFS NAS - since then I have created a Gluster volume that spans the 3 hosts and I'm putting a few VM's in there for higher reliability (as SAN is single point of failure) namely I'd like to put HostedEngine in there so it stays up no matter what and can help report if issues occur (network issue to NAS, NAS dies etc etc) Looking through other posts and documentation, there's no real way to move the HostedEngine storage, is this correct? The solution I've seen is to backup the hosted engine DB, blow it away, and re-deploy it from the .backup file configuring it to the new storage domain in the deploy script - is this the only process? How likely is this to fail? Is it likely that all VM's and settings will be picked straight back up and continue to operate like normal? I dont have a test setup to play around with atm so just trying to gauge confidence in such a solution. Thanks, Joe ___ Users mailing list -- mailto:users@ovirt.org To unsubscribe send an email to mailto:users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZTO6Q3UTEDJCDAMNFX47UR6WSM255TJ3/ ___ Users mailing list -- mailto:users@ovirt.org To unsubscribe send an email to mailto:users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JDV66LBB6WUIB3HBPVM3RUOCOO5HEB75/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DZG7UPYTUSVEIUAF244VDMXU7CZ3N2EE/___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VIENPE5YKFPA5FETQAYW6VA4EKG7HWQU/
[ovirt-users] Re: Disk move succeed but didn't move content
On Sun, Dec 1, 2019 at 1:32 AM jplor...@gmail.com wrote: > Thanks but it didn't work, seems that all data in the disk is gone. > As I still have the original storage domain, I'll see to import the vms > back. I can't add it as a posix fs, don't know what I'm doing wrong. The > ovirt docs are quite few, maybe after this I'll write something to add to > the site. > Have you referred to RHV administration guide? https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/sect-importing_existing_storage_domains > Any other ideas are welcome > Regards > > El sáb., 30 de noviembre de 2019 3:19 p. m., Amit Bawer > escribió: > >> Are you able to extend the disks to 1GB+ size ? >> >>- Go to “Virtual Machines” tab and select virtual machine >>- Go to “Disks” sub tab and select disk >>- Click on “Edit”, pay attention that if disk is locked or VM has >>other status than “UP”, “PAUSED”, “DOWN” or “SUSPENDED”, editing is not >>allowed so “Edit” option is grayed out. >>- Use “Extend Size By(GB)” field to insert the size in GB which >>should be added to the existing size >> >> >> On Fri, Nov 29, 2019 at 3:48 AM Juan Pablo Lorier >> wrote: >> >>> Hi, >>> >>> I've a fresh new install of ovirt 4.3 and tried to import an gluster >>> vmstore. I managed to import via NFS the former data domain. The problem >>> is that when I moved the disks of the vms to the new ISCSI data domain, >>> I got a warning that sparse disk type will be converted to qcow2 disks, >>> and after accepting, the disks were moved with no error. >>> >>> The problem is that the disks now figure as <1Gb size instead of the >>> original size and thus, the vms fail to start. >>> >>> Is there any way to recover those disks? I have no backup of the vms :-( >>> >>> Regards >>> ___ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/YKK2HIGPFJUZBS5KQHIIWCP5OGC3ZYVY/ >>> >> ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/46E4EPYBIL3MAGVV4KDYFXOBNGFDTZIG/
