[ovirt-users] Re: Emergency :/ No VMs starting

On February 6, 2020 6:04:58 PM GMT+02:00, Jayme wrote: >Appreciate the updates you've been posting. It's concerning to me as a >Gluster user as well. It would be nice to figure out what happened >here. > >On Thu, Feb 6, 2020 at 11:43 AM Christian Reiss > >wrote: > >> Hey, >> >> For prosperity:

[ovirt-users] Re: Emergency :/ No VMs starting

Appreciate the updates you've been posting. It's concerning to me as a Gluster user as well. It would be nice to figure out what happened here. On Thu, Feb 6, 2020 at 11:43 AM Christian Reiss wrote: > Hey, > > For prosperity: Sadly the only way to fix this was to re-init (wipe) > gluster and

[ovirt-users] Re: Emergency :/ No VMs starting

Hey, For prosperity: Sadly the only way to fix this was to re-init (wipe) gluster and start from scratch. -Chris. On 03/02/2020 19:23, Strahil Nikolov wrote: On February 3, 2020 2:29:55 PM GMT+02:00, Christian Reiss wrote: Ugh, disregarding off all previous stamenets: new findinds:

[ovirt-users] Re: Emergency :/ No VMs starting

On February 3, 2020 2:29:55 PM GMT+02:00, Christian Reiss wrote: >Ugh, > >disregarding off all previous stamenets: > >new findinds: vdsm user can NOT read files larger than 64mb. Root can. > >[vdsm@node02:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 >68 >; do dd if=/dev/urandom

[ovirt-users] Re: Emergency :/ No VMs starting

Ugh, disregarding off all previous stamenets: new findinds: vdsm user can NOT read files larger than 64mb. Root can. [vdsm@node02:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 68 ; do dd if=/dev/urandom of=file-$i bs=1M count=$i ; done

[ovirt-users] Re: Emergency :/ No VMs starting

Further findings: - modified data gets written to local node, not across gluster. - vdsm user can create _new_ files on the cluster, this gets synced immediatly. - vdsm can modify, across all nodes newly created files, changes apply immediately. I think vdsm user can not modify already

[ovirt-users] Re: Emergency :/ No VMs starting

Hey, I think I am barking up the right tree with something (else) here; Note the timestamps & id's: dd'ing a disk image as vdsm user, try 1:

[ovirt-users] Re: Emergency :/ No VMs starting

On February 3, 2020 12:25:05 PM GMT+02:00, Christian Reiss wrote: >Hey, > >here is one more thing: The issue we had some time ago might (just >might) be the culprit. We Copied the one gluster file over to the other > >nodes. The one correct node which we took down yesterday is node01, >which

[ovirt-users] Re: Emergency :/ No VMs starting

Hey, here is one more thing: The issue we had some time ago might (just might) be the culprit. We Copied the one gluster file over to the other nodes. The one correct node which we took down yesterday is node01, which has more metadata to said file: [root@node01:~] # getfattr -m . -d -e hex

[ovirt-users] Re: Emergency :/ No VMs starting

On February 3, 2020 5:06:16 AM GMT+02:00, Christian Reiss wrote: >Hey, > >they're all in this form: > >[root@node03:[..]/images/6113f079-fd28-4165-a807-61bb7625cd48] # l >total 49G >drwxr-xr-x. 2 vdsm kvm 8.0K Jan 29 23:02 . >drwxr-xr-x. 46 vdsm kvm 8.0K Feb 2 23:18 .. >-rw-rw. 1 vdsm kvm

[ovirt-users] Re: Emergency :/ No VMs starting

Hey, they're all in this form: [root@node03:[..]/images/6113f079-fd28-4165-a807-61bb7625cd48] # l total 49G drwxr-xr-x. 2 vdsm kvm 8.0K Jan 29 23:02 . drwxr-xr-x. 46 vdsm kvm 8.0K Feb 2 23:18 .. -rw-rw. 1 vdsm kvm 50G Jan 29 02:02 83f7942f-c74e-4bc4-a816-09988e7ab308 -rw-rw. 1

[ovirt-users] Re: Emergency :/ No VMs starting

Check the contents of these directories: [root@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net :_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images] # l total 345K drwxr-xr-x. 46 vdsm kvm 8.0K Feb 2 23:18 . drwxr-xr-x. 5 vdsm kvm 64

[ovirt-users] Re: Emergency :/ No VMs starting

Hey, it was _while_ placing the host _into_ maintenance, to be precise. I restarted the volumes and even each machine and the entire cluster to no avail. I am currently migrating the disk images out of ovirt into openvz/kvm to get them running. The copied disk images are flawless and

[ovirt-users] Re: Emergency :/ No VMs starting

I checked my HCI cluster and those permissions seem to match what I'm seeing. Since there's no VMs running currently have you tried restarting the gluster volumes as well as the glusterd service? I'm not sure what would have caused this with one host placed in maintenance. On Sun, Feb 2, 2020 at

[ovirt-users] Re: Emergency :/ No VMs starting

Thanks for replying. /gluster_bricks/ssd_storage/ssd_storage/.shard is root:root 0660, [root@node03:/gluster_bricks/ssd_storage/ssd_storage] # l total 5.8M drwxr-xr-x. 5 vdsm kvm98 Feb 3 02:31 . drwxr-xr-x. 3 root root 25 Jan 9 15:49 .. drwxr-xr-x. 5 vdsm kvm64 Feb 3 00:31

[ovirt-users] Re: Emergency :/ No VMs starting

The log appears to indicate that there may be a permissions issue. What is the ownership and permissions on your gluster brick dirs and mounts? On Sun, Feb 2, 2020 at 8:21 PM Christian Reiss wrote: > Hey folks, > > oh Jesus. 3-Way HCI. Gluster w/o any issues: > >

[ovirt-users] Re: Emergency :/ No VMs starting

I forgot the additional logs. Please guys, any help... (insert scream here). On 03/02/2020 01:20, Christian Reiss wrote: Hey folks, oh Jesus. 3-Way HCI. Gluster w/o any issues: [root@node01:/var/log/glusterfs] # gluster vol info  ssd_storage Volume Name: ssd_storage Type: Replicate Volume