subject:"\[ovirt\-users\] Re\: oVirt \+ Gluster"

Most peobably that VM has a very old snapshot. Sadly, 'deletion' ( merging 
snapahot to base disk) takes extra space.
You have to find temporary storage that can be added to the volume , in order 
to delete that snapahot and free enough space.
Best Regards,Strahil Nikolov
 
 
  On Mon, Jun 21, 2021 at 20:42, José Ferradeira via Users 
wrote:   ___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GDF3V6AAMAU43OFAGDFENVXVFTUI7ZLJ/
  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PMYEJYZJ4SUBK2QMOLCQYYVHAPWL7WQC/

[ovirt-users] Re: oVirt + Gluster issues

If you enabled sharding, don't disable it any more.My previous comment was just 
a statement and not a recommendation.
Based on the output, the /home2/brick2  is part of the root.
Obviosly, gluster thinks that the brick is full and you got very few options:- 
extend any brick or add more bricks to the volume- shutdown all VMs on that 
volume, reduce the minimum reserved space in gluster (gluster volume set data1 
cluster.min-free-disk 0%) and then storage migrate the disk to a bigger 
datastore- shitdown all VMs on that volume(datastore),reduce the minimum 
reserved space in gluster (gluster volume set data1 cluster.min-free-disk 0%) 
and then delete unnecessary data (like snapshots you don't need any more).
The first option is the most reliable. Once gluster has some free space to 
operate, you will be able to delete (consolidate) VM snapshots or to delete VMs 
that are less important (for example sandboxes, test VMs, etc). Yet, paused VMs 
will resume automatically and will continue to fill in Gluster which could lead 
to the same situation - so just power them off (ugly but most probably 
necessary).

 Best Regards,Strahil Nikolov
 
  On Mon, Jun 14, 2021 at 20:21, supo...@logicworks.pt 
wrote:   # df -h /home/brick1 
Sist.fichs   Tama  Ocup Livre Uso% Montado em 
/dev/mapper/cl-home  1,8T  1,8T   18G 100% /home
 
# df -h /home2/brick2  
Sist.fichs   Tama  Ocup Livre Uso% Montado em 
/dev/mapper/cl-root   50G   28G   23G  56% /
 

If sharding is enable the restore pauses the VM with unknown storage error

Thanks
José

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Segunda-feira, 14 De Junho de 2021 17:14:15
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

And what is the status of the bricks:
df -h /home/brick1 /home2/brick2
When sharding is not enabled, the qcow2 disks cannot be spread between the 
bricks.

Best Regards,Strahil Nikolov 
 
# gluster volume info data1   
  
Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on
 

# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port  RDMA Port  Online  Pid 
-- 
Brick gs.domain.pt:/home/brick1    49153 0  Y   1824862 
Brick gs.domain.pt:/home2/brick2   49154 0  Y   1824880 
  
Task Status of Volume data1 
-- 
There are no active volume tasks
 
# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed.
 

# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs   Tama  Ocup Livre Uso% Montado em 
gs.domain.pt:/data1  1,9T  1,8T   22G  99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1
 
thanks
José

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Can you provode the output of:gluster volume info VOLUME gluster volume status 
VOLUMEgluster volume heal VOLUME info summary
df -h /rhev/data-center/mnt/glusterSD/:_

In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume.If the VM has thin qcow2 disks, it 
will grow slowly till it reaches its maximum size or till the volume space is 
finished.
Best Regards,Strahil Nikolov


Well, I have one brick without space, 1.8TB.In fact I don't know why, because I 
only have one VM on that domain storage with less the 1TB.
When I try to start the VM I get this error:

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b

[ovirt-users] Re: oVirt + Gluster issues

# gluster volume info data1 

Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on 


# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port RDMA Port Online Pid 
-- 
Brick gs.domain.pt:/home/brick1 49153 0 Y 1824862 
Brick gs.domain.pt:/home2/brick2 49154 0 Y 1824880 

Task Status of Volume data1 
-- 
There are no active volume tasks 

# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed. 


# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs Tama Ocup Livre Uso% Montado em 
gs.domain.pt:/data1 1,9T 1,8T 22G 99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1 

thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Can you provode the output of: 
gluster volume info VOLUME 
gluster volume status VOLUME 
gluster volume heal VOLUME info summary 
df -h /rhev/data-center/mnt/glusterSD/:_ 


In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume. 
If the VM has thin qcow2 disks, it will grow slowly till it reaches its maximum 
size or till the volume space is finished. 

Best Regards, 
Strahil Nikolov 




Well, I have one brick without space, 1.8TB. 
In fact I don't know why, because I only have one VM on that domain storage 
with less the 1TB. 
When I try to start the VM I get this error: 

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I'm stuck in here 

Thanks 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users"  
Cc: "Alex McWhirter" , "José Ferradeira via Users" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

So, 

how is it going ? 
Do you have space ? 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov 
 wrote: 
You need to use thick VM disks on Gluster, which is the default behavior for a 
long time. 
Also, check all bricks' free space. Most probably you are out of space on one 
of the bricks (term for server + mountpoint combination). 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
 wrote: 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
 




BQ_END



BQ_END


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C3BQ36NDBBT3LDEG4MKCCKP5SEVYGFGH/

[ovirt-users] Re: oVirt + Gluster issues

# df -h /home/brick1 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-home 1,8T 1,8T 18G 100% /home 

# df -h /home2/brick2 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-root 50G 28G 23G 56% / 


If sharding is enable the restore pauses the VM with unknown storage error 

Thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 17:14:15 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

And what is the status of the bricks: 

df -h /home/brick1 /home2/brick2 

When sharding is not enabled, the qcow2 disks cannot be spread between the 
bricks. 

Best Regards, 
Strahil Nikolov 





# gluster volume info data1 

Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on 


# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port RDMA Port Online Pid 
-- 
Brick gs.domain.pt:/home/brick1 49153 0 Y 1824862 
Brick gs.domain.pt:/home2/brick2 49154 0 Y 1824880 

Task Status of Volume data1 
-- 
There are no active volume tasks 

# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed. 


# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs Tama Ocup Livre Uso% Montado em 
gs.domain.pt:/data1 1,9T 1,8T 22G 99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1 

thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Can you provode the output of: 
gluster volume info VOLUME 
gluster volume status VOLUME 
gluster volume heal VOLUME info summary 
df -h /rhev/data-center/mnt/glusterSD/:_ 


In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume. 
If the VM has thin qcow2 disks, it will grow slowly till it reaches its maximum 
size or till the volume space is finished. 

Best Regards, 
Strahil Nikolov 

BQ_BEGIN


Well, I have one brick without space, 1.8TB. 
In fact I don't know why, because I only have one VM on that domain storage 
with less the 1TB. 
When I try to start the VM I get this error: 

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I'm stuck in here 

Thanks 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users"  
Cc: "Alex McWhirter" , "José Ferradeira via Users" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

So, 

how is it going ? 
Do you have space ? 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov 
 wrote: 
You need to use thick VM disks on Gluster, which is the default behavior for a 
long time. 
Also, check all bricks' free space. Most probably you are out of space on one 
of the bricks (term for server + mountpoint combination). 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
 wrote: 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
 




BQ_END



BQ_E

[ovirt-users] Re: oVirt + Gluster issues

Can you provode the output of:gluster volume info VOLUME gluster volume status 
VOLUMEgluster volume heal VOLUME info summary
df -h /rhev/data-center/mnt/glusterSD/:_

In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume.If the VM has thin qcow2 disks, it 
will grow slowly till it reaches its maximum size or till the volume space is 
finished.
Best Regards,Strahil Nikolov
 
Well, I have one brick without space, 1.8TB. In fact I don't know why, 
because I only have one VM on that domain storage with less the 1TB.
When I try to start the VM I get this error:

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device.
I'm stuck in here
Thanks

José
De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, "José Ferradeira via Users" 
Cc: "Alex McWhirter" , "José Ferradeira via Users" 

Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

So,
how is it going ?Do you have space ?
Best Regards,Strahil Nikolov

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov wrote:You 
need to use thick VM disks on Gluster, which is the default behavior for a long 
time.Also, check all bricks' free space. Most probably you are out of space on 
one of the bricks (term for server + mountpoint combination).
Best Regards,Strahil Nikolov

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
wrote:___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/



  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FXONFEZUGEYRMBPUTJ6GNTTGA4XTXLZL/

[ovirt-users] Re: oVirt + Gluster issues

So,
how is it going ?Do you have space ?
Best Regards,Strahil Nikolov 
 
  On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov wrote:  
 You need to use thick VM disks on Gluster, which is the default behavior for a 
long time.Also, check all bricks' free space. Most probably you are out of 
space on one of the bricks (term for server + mountpoint combination).
Best Regards,Strahil Nikolov 
 
  On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
wrote:   ___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
  
  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NITZLI7T4P7UFJCYEKOBXDQ5JELSFBU6/

[ovirt-users] Re: oVirt + Gluster issues

Do you mean, in manage domain - Critical Space Action Blocker (GB) - change 5 
for 1 ? 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Quarta-feira, 16 De Junho de 2021 4:50:42 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Did you reduce the minimum free space option in gluster prior removing of the 
snapshot (and it's failure) ? 

Best Regards, 
Strahil Nikolov 




On Wed, Jun 16, 2021 at 0:35, supo...@logicworks.pt 
 wrote: 
Yes, there is one snapshot but I cannot remove it: 
Error while executing action: Cannot remove Disk Snapshot. Low disk space on 
Storage Domain DATA1. 

Regards 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 18:45:49 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Did you check for snapshots ? 

You can check the contents of the /rhev... mount point. 

Best Regards, 
Strahil Nikolov 






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain. 



 
De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

You will need to free some space. 
Check my previous e-mail. 

Bedt Regards, 
Strahil Nikolov 




___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BH3AH3HFPZLN2GUXULBS3B4VCSPJUW6I/

[ovirt-users] Re: oVirt + Gluster issues

I just change cluster.min-free-disk to 5% 
but still get the message: Error while executing action: Cannot remove Disk 
Snapshot. Low disk space on Storage Domain DATA1. 

# gluster volume get data1 all|grep cluster.min-free-disk 
cluster.min-free-disk 5% 


De: supo...@logicworks.pt 
Para: "Strahil Nikolov"  
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Quarta-feira, 16 De Junho de 2021 16:31:46 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Do you mean, in manage domain - Critical Space Action Blocker (GB) - change 5 
for 1 ? 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Quarta-feira, 16 De Junho de 2021 4:50:42 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Did you reduce the minimum free space option in gluster prior removing of the 
snapshot (and it's failure) ? 

Best Regards, 
Strahil Nikolov 




On Wed, Jun 16, 2021 at 0:35, supo...@logicworks.pt 
 wrote: 
Yes, there is one snapshot but I cannot remove it: 
Error while executing action: Cannot remove Disk Snapshot. Low disk space on 
Storage Domain DATA1. 

Regards 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 18:45:49 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Did you check for snapshots ? 

You can check the contents of the /rhev... mount point. 

Best Regards, 
Strahil Nikolov 






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain. 



 
De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

You will need to free some space. 
Check my previous e-mail. 

Bedt Regards, 
Strahil Nikolov 




___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/65PDKRG7QZ7YBZGD3JBDTMKCGOESZOV3/

[ovirt-users] Re: oVirt + Gluster issues

You need to use thick VM disks on Gluster, which is the default behavior for a 
long time.Also, check all bricks' free space. Most probably you are out of 
space on one of the bricks (term for server + mountpoint combination).
Best Regards,Strahil Nikolov 
 
  On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
wrote:   ___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJE33UNL5DTLZ6KLRXDDKSWM63M67ONL/

[ovirt-users] Re: oVirt + Gluster issues

# df -h /home/brick1 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-home 1,8T 1,8T 18G 100% /home 

# df -h /home2/brick2 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-root 50G 28G 23G 56% / 


If sharding is enable the restore pauses the VM with unknown storage error 

Thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 17:14:15 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

And what is the status of the bricks: 

df -h /home/brick1 /home2/brick2 

When sharding is not enabled, the qcow2 disks cannot be spread between the 
bricks. 

Best Regards, 
Strahil Nikolov 





# gluster volume info data1 

Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on 


# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port RDMA Port Online Pid 
-- 
Brick gs.domain.pt:/home/brick1 49153 0 Y 1824862 
Brick gs.domain.pt:/home2/brick2 49154 0 Y 1824880 

Task Status of Volume data1 
-- 
There are no active volume tasks 

# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed. 


# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs Tama Ocup Livre Uso% Montado em 
gs.domain.pt:/data1 1,9T 1,8T 22G 99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1 

thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Can you provode the output of: 
gluster volume info VOLUME 
gluster volume status VOLUME 
gluster volume heal VOLUME info summary 
df -h /rhev/data-center/mnt/glusterSD/:_ 


In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume. 
If the VM has thin qcow2 disks, it will grow slowly till it reaches its maximum 
size or till the volume space is finished. 

Best Regards, 
Strahil Nikolov 

BQ_BEGIN


Well, I have one brick without space, 1.8TB. 
In fact I don't know why, because I only have one VM on that domain storage 
with less the 1TB. 
When I try to start the VM I get this error: 

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I'm stuck in here 

Thanks 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users"  
Cc: "Alex McWhirter" , "José Ferradeira via Users" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

So, 

how is it going ? 
Do you have space ? 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov 
 wrote: 
You need to use thick VM disks on Gluster, which is the default behavior for a 
long time. 
Also, check all bricks' free space. Most probably you are out of space on one 
of the bricks (term for server + mountpoint combination). 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
 wrote: 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
 




BQ_END



BQ_E

[ovirt-users] Re: oVirt + Gluster issues

Your last e-mail was empty.

Best Regards,Strahil Nikolov___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YPHAFLN4B6QQFNKQKXQCEG2HJPI7N6RA/

[ovirt-users] Re: oVirt + Gluster issues

Yes, there is one snapshot but I cannot remove it: 
Error while executing action: Cannot remove Disk Snapshot. Low disk space on 
Storage Domain DATA1. 

Regards 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 18:45:49 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Did you check for snapshots ? 

You can check the contents of the /rhev... mount point. 

Best Regards, 
Strahil Nikolov 






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain. 



 
De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

You will need to free some space. 
Check my previous e-mail. 

Bedt Regards, 
Strahil Nikolov 
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TISW3B7SHQXOJDJN7S5RIT4SC27KDHFX/

[ovirt-users] Re: oVirt + Gluster issues

Did you reduce the minimum free space option in gluster prior removing of the 
snapshot (and it's failure) ?
Best Regards,Strahil Nikolov
 
 
  On Wed, Jun 16, 2021 at 0:35, supo...@logicworks.pt 
wrote:   Yes, there is one snapshot but I cannot remove it:
Error while executing action: Cannot remove Disk Snapshot. Low disk space on 
Storage Domain DATA1.
Regards
José

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Terça-feira, 15 De Junho de 2021 18:45:49
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Did you check for snapshots ?

You can check the contents of the /rhev... mount point.

Best Regards,
Strahil Nikolov






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain.




De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

You will need to free some space.
Check my previous e-mail.

Bedt Regards,
Strahil Nikolov
  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G5BAVGPI3W562S6SIKDJAE2HRKCW3L7A/

[ovirt-users] Re: oVirt + Gluster issues

And what is the status of the bricks:
df -h /home/brick1 /home2/brick2
When sharding is not enabled, the qcow2 disks cannot be spread between the 
bricks.

Best Regards,Strahil Nikolov 
 
# gluster volume info data1   
  
Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on
 

# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port  RDMA Port  Online  Pid 
-- 
Brick gs.domain.pt:/home/brick1    49153 0  Y   1824862 
Brick gs.domain.pt:/home2/brick2   49154 0  Y   1824880 
  
Task Status of Volume data1 
-- 
There are no active volume tasks
 
# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed.
 

# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs   Tama  Ocup Livre Uso% Montado em 
gs.domain.pt:/data1  1,9T  1,8T   22G  99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1
 
thanks
José

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Can you provode the output of:gluster volume info VOLUME gluster volume status 
VOLUMEgluster volume heal VOLUME info summary
df -h /rhev/data-center/mnt/glusterSD/:_

In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume.If the VM has thin qcow2 disks, it 
will grow slowly till it reaches its maximum size or till the volume space is 
finished.
Best Regards,Strahil Nikolov


Well, I have one brick without space, 1.8TB.In fact I don't know why, because I 
only have one VM on that domain storage with less the 1TB.
When I try to start the VM I get this error:

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device.
I'm stuck in here
Thanks

José
De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, "José Ferradeira via Users" 
Cc: "Alex McWhirter" , "José Ferradeira via Users" 

Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

So,
how is it going ?Do you have space ?
Best Regards,Strahil Nikolov

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov wrote:You 
need to use thick VM disks on Gluster, which is the default behavior for a long 
time.Also, check all bricks' free space. Most probably you are out of space on 
one of the bricks (term for server + mountpoint combination).
Best Regards,Strahil Nikolov

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
wrote:___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/





  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FN7IYHQBYPK3GKB3PCHT6HUSY5RH7AHC/

[ovirt-users] Re: oVirt + Gluster issues

I just free every space I could. That's the only VM in that storage domain. 



De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

You will need to free some space. 
Check my previous e-mail. 

Bedt Regards, 
Strahil Nikolov 
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GDF3V6AAMAU43OFAGDFENVXVFTUI7ZLJ/

[ovirt-users] Re: oVirt + Gluster issues

Well, I have one brick without space, 1.8TB. 
In fact I don't know why, because I only have one VM on that domain storage 
with less the 1TB. 
When I try to start the VM I get this error: 

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I'm stuck in here 

Thanks 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users"  
Cc: "Alex McWhirter" , "José Ferradeira via Users" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

So, 

how is it going ? 
Do you have space ? 

Best Regards, 
Strahil Nikolov 




On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov 
 wrote: 
You need to use thick VM disks on Gluster, which is the default behavior for a 
long time. 
Also, check all bricks' free space. Most probably you are out of space on one 
of the bricks (term for server + mountpoint combination). 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
 wrote: 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
 




BQ_END


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4SXTSMNTD7DF5MAIFSEZP5LOLTV3C24B/

[ovirt-users] Re: oVirt + Gluster issues

# df -h /home/brick1 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-home 1,8T 1,8T 18G 100% /home 

# df -h /home2/brick2 
Sist.fichs Tama Ocup Livre Uso% Montado em 
/dev/mapper/cl-root 50G 28G 23G 56% / 


If sharding is enable the restore pauses the VM with unknown storage error 

Thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 17:14:15 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

And what is the status of the bricks: 

df -h /home/brick1 /home2/brick2 

When sharding is not enabled, the qcow2 disks cannot be spread between the 
bricks. 

Best Regards, 
Strahil Nikolov 




# gluster volume info data1 

Volume Name: data1 
Type: Distribute 
Volume ID: d7eb2c38-2707-4774-9873-a7303d024669 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: gs.domain.pt:/home/brick1 
Brick2: gs.domain.pt:/home2/brick2 
Options Reconfigured: 
nfs.disable: on 
transport.address-family: inet 
storage.fips-mode-rchecksum: on 
storage.owner-uid: 36 
storage.owner-gid: 36 
cluster.min-free-disk: 10% 
performance.quick-read: off 
performance.read-ahead: off 
performance.io-cache: off 
performance.low-prio-threads: 32 
network.remote-dio: enable 
cluster.eager-lock: enable 
cluster.quorum-type: auto 
cluster.server-quorum-type: server 
cluster.data-self-heal-algorithm: full 
cluster.locking-scheme: granular 
cluster.shd-wait-qlength: 1 
features.shard: off 
user.cifs: off 
cluster.choose-local: off 
client.event-threads: 4 
server.event-threads: 4 
performance.client-io-threads: on 


# gluster volume status data1 
Status of volume: data1 
Gluster process TCP Port RDMA Port Online Pid 
-- 
Brick gs.domain.pt:/home/brick1 49153 0 Y 1824862 
Brick gs.domain.pt:/home2/brick2 49154 0 Y 1824880 

Task Status of Volume data1 
-- 
There are no active volume tasks 

# gluster volume heal data1 info summary 
This command is supported for only volumes of replicate/disperse type. Volume 
data1 is not of type replicate/disperse 
Volume heal failed. 


# df -h /rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/ 
Sist.fichs Tama Ocup Livre Uso% Montado em 
gs.domain.pt:/data1 1,9T 1,8T 22G 99% 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1 

thanks 
José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 14:54:41 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

Can you provode the output of: 
gluster volume info VOLUME 
gluster volume status VOLUME 
gluster volume heal VOLUME info summary 
df -h /rhev/data-center/mnt/glusterSD/:_ 


In pure replica volumes , the bricks should be of the same size. If not - the 
smallest one defines the size of the volume. 
If the VM has thin qcow2 disks, it will grow slowly till it reaches its maximum 
size or till the volume space is finished. 

Best Regards, 
Strahil Nikolov 

BQ_BEGIN


Well, I have one brick without space, 1.8TB. 
In fact I don't know why, because I only have one VM on that domain storage 
with less the 1TB. 
When I try to start the VM I get this error: 

VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I'm stuck in here 

Thanks 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users"  
Cc: "Alex McWhirter" , "José Ferradeira via Users" 
 
Enviadas: Segunda-feira, 14 De Junho de 2021 7:21:09 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 

So, 

how is it going ? 
Do you have space ? 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Thu, Jun 10, 2021 at 18:19, Strahil Nikolov 
 wrote: 
You need to use thick VM disks on Gluster, which is the default behavior for a 
long time. 
Also, check all bricks' free space. Most probably you are out of space on one 
of the bricks (term for server + mountpoint combination). 

Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Wed, Jun 9, 2021 at 12:41, José Ferradeira via Users 
 wrote: 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/
 




BQ_END



BQ_E

[ovirt-users] Re: oVirt + Gluster issues

Exactly that one. Sadly, I got no clue how big is your snapshot(s). Maybe even 
with those extra Gigs, it's still too much to merge it.
If you have any other VMs there, try to move their disks away in order to 
release more space.
Otherwise, you will have to find another brick to extend the volume.
How many snapshots do you see in oVirt ? Maybe you have more than one snapshot.
Best Regards,Strahil Nikolov
 
 
  On Wed, Jun 16, 2021 at 21:22, supo...@logicworks.pt 
wrote:   I just change cluster.min-free-disk to 5%
but still get the message: Error while executing action: Cannot remove Disk 
Snapshot. Low disk space on Storage Domain DATA1.

# gluster volume get data1 all|grep cluster.min-free-disk
cluster.min-free-disk 5% 
De: supo...@logicworks.pt
Para: "Strahil Nikolov" 
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Quarta-feira, 16 De Junho de 2021 16:31:46
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Do you mean, in manage domain - Critical Space Action Blocker (GB) - change 5 
for 1 ?

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Quarta-feira, 16 De Junho de 2021 4:50:42
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Did you reduce the minimum free space option in gluster prior removing of the 
snapshot (and it's failure) ?
Best Regards,Strahil Nikolov
 

On Wed, Jun 16, 2021 at 0:35, supo...@logicworks.pt 
wrote:Yes, there is one snapshot but I cannot remove it:
Error while executing action: Cannot remove Disk Snapshot. Low disk space on 
Storage Domain DATA1.
Regards
José

De: "Strahil Nikolov" 
Para: supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Terça-feira, 15 De Junho de 2021 18:45:49
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

Did you check for snapshots ?

You can check the contents of the /rhev... mount point.

Best Regards,
Strahil Nikolov






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain.




De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

You will need to free some space.
Check my previous e-mail.

Bedt Regards,
Strahil Nikolov


  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BULE25KWY7GHT4X3TD4M34HRTN7KJNTG/

[ovirt-users] Re: oVirt + Gluster issues

Did you check for snapshots ?

You can check the contents of the /rhev... mount point.

Best Regards,
Strahil Nikolov






В вторник, 15 юни 2021 г., 18:49:41 ч. Гринуич+3,  
написа: 





I just free every space I could. That's the only VM in that storage domain.




De: "Strahil Nikolov" 
Para: supo...@logicworks.pt, supo...@logicworks.pt
Cc: "José Ferradeira via Users" , "Alex McWhirter" 

Enviadas: Terça-feira, 15 De Junho de 2021 16:04:56
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues

You will need to free some space.
Check my previous e-mail.

Bedt Regards,
Strahil Nikolov
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/77DSAYPBTW2ASMDZL7SR475NV3MLDQA3/

[ovirt-users] Re: oVirt + Gluster issues

You will need to free some space.Check my previous e-mail.
Bedt Regards,Strahil Nikolov___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HZV5N23FVRXG6IWV4XCAVR7MDJEU46LV/

[ovirt-users] Re: oVirt + Gluster issues

2021-06-09 Thread José Ferradeira via Users

The gluster storage domain, now is a mess, I cannot run a VM, always get this 
error: 
VM webmail.domain.pt-3 is down with error. Exit message: Unable to set XATTR 
trusted.libvirt.security.selinux on 
/rhev/data-center/mnt/glusterSD/gs1.domain.pt:_data1/d680d289-bcaa-46f2-b464-4d06d37ec1d3/images/5167f58d-68c9-475f-8b88-f278b7d4ef65/9b34eff0-c9a4-48e1-8ea7-87ad66a8736c:
 No space left on device. 

I had a second brick and have 1.5TB free, I don't know why it says "No space 
left on device" 

Also, I cannot use the iso files I have in this storage domain. 

Regards 
José 



De: "José Ferradeira via Users"  
Para: "Alex McWhirter"  
Cc: "Strahil Nikolov" , "José Ferradeira via Users" 
 
Enviadas: Terça-feira, 8 De Junho de 2021 22:47:25 
Assunto: [ovirt-users] Re: oVirt + Gluster issues 

This is a glusterfs on top of CentOS 8.3, using LVM 


De: "Alex McWhirter"  
Para: "Strahil Nikolov"  
Cc: supo...@logicworks.pt, "José Ferradeira via Users"  
Enviadas: Terça-feira, 8 De Junho de 2021 17:48:48 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 



I've run into a similar problem when using VDO + LVM + XFS stacks, also with 
ZFS. 

If you're trying to use ZFS on 4.4, my recommendation is don't. You have to run 
the testing branch at minimum, and quiet a few things just don't work. 




As for VDO, i ran into this issue when using VDO and a NVME for LVM caching of 
the thin pool, VDO would throw a fit and under high load scenario, VM's would 
regularly pause. 

VDO with no cache was fine however, seems to be related to mixing device types 
/ block sizes (even if you override block sizes). 




Not sure if that helps. 

On 2021-06-08 12:26, Strahil Nikolov via Users wrote: 


Maybe the shard xlator cannot cope with the speed of the shard creation speed. 
Are you using preallocated disks on the Zimbra VM ? 
Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Tue, Jun 8, 2021 at 17:57, José Ferradeira via Users 
 wrote: 
Hello, 
running ovirt 4.4.4.7-1.el8 and gluster 8.3. 
When i performe a restore of Zimbra Collaboration Email with features.shard on, 
the VM pauses with an unknown storage error. 
When I performe a restore of Zimbra Collaboration Email with features.shard 
off, it fills all the gluster storage domain disks. 
With older versions of gluster and ovirt the same happens. If I use a NFS 
storage domain it runs OK. 
-- 

Jose Ferradeira 
http://www.logicworks.pt 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QGBFSIHHTDOTTOWFWQKZFQMD56YWHTPZ/
 




___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MOJJZJG7LCGHDIYULF5572L52JE53T6D/
 

BQ_END





___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QDEHBKD4OVRK3MC6ZFGFBNPVPYMK2VXI/
 
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SC6RR2YC35OOOUEDEVSVCQ7RMW56DCSJ/

[ovirt-users] Re: oVirt + Gluster issues

2021-06-08 Thread José Ferradeira via Users

This is a glusterfs on top of CentOS 8.3, using LVM 


De: "Alex McWhirter"  
Para: "Strahil Nikolov"  
Cc: supo...@logicworks.pt, "José Ferradeira via Users"  
Enviadas: Terça-feira, 8 De Junho de 2021 17:48:48 
Assunto: Re: [ovirt-users] Re: oVirt + Gluster issues 



I've run into a similar problem when using VDO + LVM + XFS stacks, also with 
ZFS. 

If you're trying to use ZFS on 4.4, my recommendation is don't. You have to run 
the testing branch at minimum, and quiet a few things just don't work. 




As for VDO, i ran into this issue when using VDO and a NVME for LVM caching of 
the thin pool, VDO would throw a fit and under high load scenario, VM's would 
regularly pause. 

VDO with no cache was fine however, seems to be related to mixing device types 
/ block sizes (even if you override block sizes). 




Not sure if that helps. 

On 2021-06-08 12:26, Strahil Nikolov via Users wrote: 


Maybe the shard xlator cannot cope with the speed of the shard creation speed. 
Are you using preallocated disks on the Zimbra VM ? 
Best Regards, 
Strahil Nikolov 


BQ_BEGIN

On Tue, Jun 8, 2021 at 17:57, José Ferradeira via Users 
 wrote: 
Hello, 
running ovirt 4.4.4.7-1.el8 and gluster 8.3. 
When i performe a restore of Zimbra Collaboration Email with features.shard on, 
the VM pauses with an unknown storage error. 
When I performe a restore of Zimbra Collaboration Email with features.shard 
off, it fills all the gluster storage domain disks. 
With older versions of gluster and ovirt the same happens. If I use a NFS 
storage domain it runs OK. 
-- 

Jose Ferradeira 
http://www.logicworks.pt 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QGBFSIHHTDOTTOWFWQKZFQMD56YWHTPZ/
 




___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MOJJZJG7LCGHDIYULF5572L52JE53T6D/
 

BQ_END




___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QDEHBKD4OVRK3MC6ZFGFBNPVPYMK2VXI/

[ovirt-users] Re: oVirt + Gluster issues

2021-06-08 Thread José Ferradeira via Users

The disks on the VM are Thin Provision 

Regards 

José 


De: "Strahil Nikolov"  
Para: supo...@logicworks.pt, "José Ferradeira via Users" , 
"oVirt Users"  
Enviadas: Terça-feira, 8 De Junho de 2021 17:26:42 
Assunto: Re: [ovirt-users] oVirt + Gluster issues 

Maybe the shard xlator cannot cope with the speed of the shard creation speed. 

Are you using preallocated disks on the Zimbra VM ? 

Best Regards, 
Strahil Nikolov 




On Tue, Jun 8, 2021 at 17:57, José Ferradeira via Users 
 wrote: 
Hello, 

running ovirt 4.4.4.7-1.el8 and gluster 8.3. 
When i performe a restore of Zimbra Collaboration Email with features.shard on, 
the VM pauses with an unknown storage error. 
When I performe a restore of Zimbra Collaboration Email with features.shard 
off, it fills all the gluster storage domain disks. 

With older versions of gluster and ovirt the same happens. If I use a NFS 
storage domain it runs OK. 



-- 

Jose Ferradeira 
http://www.logicworks.pt 
___ 
Users mailing list -- users@ovirt.org 
To unsubscribe send an email to users-le...@ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QGBFSIHHTDOTTOWFWQKZFQMD56YWHTPZ/
 




___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CJVVQZIRPVQOQQXY5T7BFZX4NYUE5N5S/

[ovirt-users] Re: oVirt + Gluster issues

2021-06-08 Thread Alex McWhirter

I've run into a similar problem when using VDO + LVM + XFS stacks, also
with ZFS.

If you're trying to use ZFS on 4.4, my recommendation is don't. You have
to run the testing branch at minimum, and quiet a few things just don't
work.

As for VDO, i ran into this issue when using VDO and a NVME for LVM
caching of the thin pool, VDO would throw a fit and under high load
scenario, VM's would regularly pause.

VDO with no cache was fine however, seems to be related to mixing device
types / block sizes (even if you override block sizes).

Not sure if that helps.

On 2021-06-08 12:26, Strahil Nikolov via Users wrote:

Maybe the shard xlator cannot cope with the speed of the shard creation speed.

Are you using preallocated disks on the Zimbra VM ?

Best Regards,
Strahil Nikolov

On Tue, Jun 8, 2021 at 17:57, José Ferradeira via Users
wrote:

Hello,

running ovirt 4.4.4.7-1.el8 and gluster 8.3.
When i performe a restore of Zimbra Collaboration Email with features.shard on, the VM pauses with an unknown storage error.
When I performe a restore of Zimbra Collaboration Email with features.shard off, it fills all the gluster storage domain disks.

With older versions of gluster and ovirt the same happens. If I use a NFS storage domain it runs OK.

Jose Ferradeira
http://www.logicworks.pt ___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QGBFSIHHTDOTTOWFWQKZFQMD56YWHTPZ/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MOJJZJG7LCGHDIYULF5572L52JE53T6D/___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JWUSPYSFOMFOHPDOPZBLYGRHYYFVCZ5H/

[ovirt-users] Re: oVirt + Gluster issues

2021-06-08 Thread Strahil Nikolov via Users

Maybe the shard xlator cannot cope with the speed of the shard creation speed.
Are you using preallocated disks on the Zimbra VM ?
Best Regards,Strahil Nikolov
 
 
  On Tue, Jun 8, 2021 at 17:57, José Ferradeira via Users 
wrote:   Hello,

running ovirt 4.4.4.7-1.el8 and gluster 8.3.
When i performe a restore of Zimbra Collaboration Email with features.shard on, 
the VM pauses with an unknown storage error.
When I performe a restore of Zimbra Collaboration Email with features.shard 
off, it fills all the gluster storage domain disks. 

With older versions of gluster and ovirt the same happens. If I use a NFS 
storage domain it runs OK.



-- 
Jose Ferradeira
http://www.logicworks.pt
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QGBFSIHHTDOTTOWFWQKZFQMD56YWHTPZ/
  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MOJJZJG7LCGHDIYULF5572L52JE53T6D/

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-24 Thread Ritesh Chikatwar

On Wed, Sep 23, 2020 at 10:16 PM Strahil Nikolov via Users 
wrote:

>
> >1) ..."I would give the engine a 'Windows'-style fix (a.k.a.
> reboot)"  >how does one restart just the oVirt-engine?
> ssh to HostedEngine VM and run one of the following:
> - reboot
> - systemctl restart ovirt-engine.service
>
> >2) I now show in shell  3 nodes, each with the one brick for data,
> vmstore, >engine (and an ISO one I am trying to make).. with one brick each
> and all >online and replicating.  But the GUI shows thor (first server
> running >engine) offline needing to be reloaded.  Now volumes show two
> bricks.. one >online one offline.  And no option to start / force restart.
> If it shows one offline brick -> you can try the "force start". You can go
> to UI -> Storage -> Volume -> select Volume -> Start and then mark "Force"
> and "OK"
>
>
> >4) To the question of "did I add third node later."  I would attach
> >deployment guide I am building ... but can't do that in this forum.  but
> >this is as simple as I can make it.  3 intel generic servers,  1 x boot
> >drive , 1 x 512GB SSD,  2 x 1TB SSD in each.  wipe all data all
> >configuration fresh Centos8 minimal install.. setup SSH setup basic
> >networking... install cockpit.. run HCI wizard for all three nodes. That
> is >all.
>
> >How many hosts do you see in oVirt ?
> >Help is appreciated.  The main concern I have is gap in what engine sees
> >and what CLI shows.  Can someone show me where to get logs?  the GUI log
> >when I try to "activate" thor server "Status of host thor was set to
> >NonOperational."  "Gluster command [] failed on server
> >."  is very unhelpful.
> Check the following services on the node:
> - glusterd.service
>
If glusterd service is running on the host check whether vdsm-client is
returning gluster host uuid. just run this on the host "vdsm-client
--gluster-enable GlusterHost uuid".

> - sanlock.service
> - supervdsmd.service
> - vdsmd.service
> - ovirt-ha-broker.service
> - ovirt-ha-agent.service
>
> Best Regards,
> Strahil Nikolov
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/YJ7L5G7NU4PQAPQDCDIMC37JCEEGAILF/
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7OKJXLUVA6G7B4NR6YVEVN3MNPMIJUXT/

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-23 Thread Gobinda Das

We do have gluster volume UI sync issue and this is fixed in ovirt-4.4.2
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1860775
Could be the same issue.

On Wed, Sep 23, 2020 at 10:16 PM Strahil Nikolov via Users 
wrote:

>
> >1) ..."I would give the engine a 'Windows'-style fix (a.k.a.
> reboot)"  >how does one restart just the oVirt-engine?
> ssh to HostedEngine VM and run one of the following:
> - reboot
> - systemctl restart ovirt-engine.service
>
> >2) I now show in shell  3 nodes, each with the one brick for data,
> vmstore, >engine (and an ISO one I am trying to make).. with one brick each
> and all >online and replicating.  But the GUI shows thor (first server
> running >engine) offline needing to be reloaded.  Now volumes show two
> bricks.. one >online one offline.  And no option to start / force restart.
> If it shows one offline brick -> you can try the "force start". You can go
> to UI -> Storage -> Volume -> select Volume -> Start and then mark "Force"
> and "OK"
>
>
> >4) To the question of "did I add third node later."  I would attach
> >deployment guide I am building ... but can't do that in this forum.  but
> >this is as simple as I can make it.  3 intel generic servers,  1 x boot
> >drive , 1 x 512GB SSD,  2 x 1TB SSD in each.  wipe all data all
> >configuration fresh Centos8 minimal install.. setup SSH setup basic
> >networking... install cockpit.. run HCI wizard for all three nodes. That
> is >all.
>
> >How many hosts do you see in oVirt ?
> >Help is appreciated.  The main concern I have is gap in what engine sees
> >and what CLI shows.  Can someone show me where to get logs?  the GUI log
> >when I try to "activate" thor server "Status of host thor was set to
> >NonOperational."  "Gluster command [] failed on server
> >."  is very unhelpful.
> Check the following services on the node:
> - glusterd.service
> - sanlock.service
> - supervdsmd.service
> - vdsmd.service
> - ovirt-ha-broker.service
> - ovirt-ha-agent.service
>
> Best Regards,
> Strahil Nikolov
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/YJ7L5G7NU4PQAPQDCDIMC37JCEEGAILF/
>


-- 


Thanks,
Gobinda
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BC3JYXVDAKANFLZQGMT6NUM6XBZNK254/

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-23 Thread Strahil Nikolov via Users


>1) ..."I would give the engine a 'Windows'-style fix (a.k.a. reboot)"  
>>how does one restart just the oVirt-engine?
ssh to HostedEngine VM and run one of the following:
- reboot
- systemctl restart ovirt-engine.service

>2) I now show in shell  3 nodes, each with the one brick for data, vmstore, 
>>engine (and an ISO one I am trying to make).. with one brick each and all 
>>online and replicating.  But the GUI shows thor (first server running 
>>engine) offline needing to be reloaded.  Now volumes show two bricks.. one 
>>online one offline.  And no option to start / force restart.
If it shows one offline brick -> you can try the "force start". You can go to 
UI -> Storage -> Volume -> select Volume -> Start and then mark "Force" and "OK"


>4) To the question of "did I add third node later."  I would attach 
>>deployment guide I am building ... but can't do that in this forum.  but 
>>this is as simple as I can make it.  3 intel generic servers,  1 x boot 
>>drive , 1 x 512GB SSD,  2 x 1TB SSD in each.  wipe all data all 
>>configuration fresh Centos8 minimal install.. setup SSH setup basic 
>>networking... install cockpit.. run HCI wizard for all three nodes. That is 
>>all.

>How many hosts do you see in oVirt ?
>Help is appreciated.  The main concern I have is gap in what engine sees >and 
>what CLI shows.  Can someone show me where to get logs?  the GUI log  >when I 
>try to "activate" thor server "Status of host thor was set to 
>>NonOperational."  "Gluster command [] failed on server >."  
>is very unhelpful.
Check the following services on the node:
- glusterd.service
- sanlock.service
- supervdsmd.service
- vdsmd.service
- ovirt-ha-broker.service
- ovirt-ha-agent.service

Best Regards,
Strahil Nikolov
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YJ7L5G7NU4PQAPQDCDIMC37JCEEGAILF/

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-23 Thread Jeremey Wise

in oVirt Engine I think I see some of the issue

When you go under volumes -> Data ->

[image: image.png]

It notes two servers..  when you choose "add brick" it says volume has 3
bricks but only two servers.

So I went back to my deployment notes and walked through setup

yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm -y

yum install -y cockpit-ovirt-dashboard vdsm-gluster ovirt-host

Last metadata expiration check: 1:59:46 ago on Wed 23 Sep 2020 06:10:46 AM
EDT.
Package cockpit-ovirt-dashboard-0.14.11-1.el8.noarch is already installed.
Package ovirt-host-4.4.1-4.el8.x86_64 is already installed.
Dependencies resolved.
=
 Package
 ArchitectureVersion
  Repository
   Size
=
Installing:
 vdsm-gluster   x86_64
 4.40.26.3-1.el8
  ovirt-4.4
  67 k
Installing dependencies:
 blivet-datanoarch
 1:3.1.0-21.el8_2
 AppStream
 238 k
 glusterfs-events   x86_64
 7.7-1.el8
  ovirt-4.4-centos-gluster7
  65 k
 glusterfs-geo-replication  x86_64
 7.7-1.el8
  ovirt-4.4-centos-gluster7
 212 k
 libblockdev-plugins-allx86_64
 2.19-12.el8
  AppStream
  62 k
 libblockdev-vdox86_64
 2.19-12.el8
  AppStream
  74 k
 python3-blivet noarch
 1:3.1.0-21.el8_2
 AppStream
 995 k
 python3-blockdev   x86_64
 2.19-12.el8
  AppStream
  79 k
 python3-bytesize   x86_64
 1.4-3.el8
  AppStream
  28 k
 python3-magic  noarch
 5.33-13.el8
  BaseOS
   45 k
 python3-pyparted   x86_64
 1:3.11.0-13.el8
  AppStream
 123 k


Dependencies resolved.
Nothing to do.
Complete!
[root@thor media]#



AKA.. something got removed from the node..


Rebooted.. as I am not sure which dependancies and services would need to
be restarted to get oVirt-engine to pick things up.


Host is now "green" .. now only errors are about gluster bricks..



On Tue, Sep 22, 2020 at 9:30 PM penguin pages 
wrote:

>
>
> eMail client with this forum is a bit .. I was told this web
> interface I could post images... as embedded ones in email get scraped
> out...  but not seeing how that is done. Seems to be txt only.
>
>
>
> 1) ..."I would give the engine a 'Windows'-style fix (a.k.a. reboot)"
> how does one restart just the oVirt-engine?
>
> 2) I now show in shell  3 nodes, each with the one brick for data,
> vmstore, engine (and an ISO one I am trying to make).. with one brick each
> and all online and replicating.   But the GUI shows thor (first server
> running engine) offline needing to be reloaded.  Now volumes show two
> bricks.. one online one offline.  And no option to start / force restart.
>
> 3) I have tried several times to try a graceful reboot to see if startup
> sequence was issue.   I tore down VLANs and bridges to make it flat 1 x 1Gb
> mgmt, 1 x 10Gb storage.   SSH between nodes is fine... copy test was
> great.   I don't think it is nodes.
>
> 4) To the question of "did I add third node later."  I would attach
> deployment guide I am building ... but can't do that in this forum.  but
> this is as simple as I can make it.  3 intel generic servers,  1 x boot
> drive , 1 x 512GB SSD,  2 x 1TB SSD in each.   wipe all data all
> configuration fresh Centos8 minimal install.. setup SSH setup basic
> networking... install cockpit.. run HCI wizard for all three nodes. That is
> all.
>
> Trying to learn and support concept of oVirt as a viable platform but
> still trying to work through learning how to root cause, kick tires, and
> debug / recover when things go down .. as they will.
>
> Help is appreciated.  The main concern I have is gap in what engine sees
>

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-22 Thread penguin pages

eMail client with this forum is a bit .. I was told this web interface
I could post images... as embedded ones in email get scraped out... but not
seeing how that is done. Seems to be txt only.

1) ..."I would give the engine a 'Windows'-style fix (a.k.a. reboot)" how
does one restart just the oVirt-engine?

2) I now show in shell 3 nodes, each with the one brick for data, vmstore,
engine (and an ISO one I am trying to make).. with one brick each and all
online and replicating. But the GUI shows thor (first server running engine)
offline needing to be reloaded. Now volumes show two bricks.. one online one
offline. And no option to start / force restart.

3) I have tried several times to try a graceful reboot to see if startup
sequence was issue. I tore down VLANs and bridges to make it flat 1 x 1Gb
mgmt, 1 x 10Gb storage. SSH between nodes is fine... copy test was great. I
don't think it is nodes.

4) To the question of "did I add third node later." I would attach deployment
guide I am building ... but can't do that in this forum. but this is as simple
as I can make it. 3 intel generic servers, 1 x boot drive , 1 x 512GB SSD, 2
x 1TB SSD in each. wipe all data all configuration fresh Centos8 minimal
install.. setup SSH setup basic networking... install cockpit.. run HCI wizard
for all three nodes. That is all.

Trying to learn and support concept of oVirt as a viable platform but still
trying to work through learning how to root cause, kick tires, and debug /
recover when things go down .. as they will.

Help is appreciated. The main concern I have is gap in what engine sees and
what CLI shows. Can someone show me where to get logs? the GUI log when I
try to "activate" thor server "Status of host thor was set to NonOperational."
"Gluster command [] failed on server ." is very unhelpful.

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

Ovirt uses the "/rhev/mnt... mountpoints.

Do you have those (for each storage domain ) ?

Here is an example from one of my nodes:
[root@ovirt1 ~]# df -hT | grep rhev
gluster1:/engine                              fuse.glusterfs  100G   19G   82G  
19% /rhev/data-center/mnt/glusterSD/gluster1:_engine
gluster1:/fast4                               fuse.glusterfs  100G   53G   48G  
53% /rhev/data-center/mnt/glusterSD/gluster1:_fast4
gluster1:/fast1                               fuse.glusterfs  100G   56G   45G  
56% /rhev/data-center/mnt/glusterSD/gluster1:_fast1
gluster1:/fast2                               fuse.glusterfs  100G   56G   45G  
56% /rhev/data-center/mnt/glusterSD/gluster1:_fast2
gluster1:/fast3                               fuse.glusterfs  100G   55G   46G  
55% /rhev/data-center/mnt/glusterSD/gluster1:_fast3
gluster1:/data                                fuse.glusterfs  2.4T  535G  1.9T  
23% /rhev/data-center/mnt/glusterSD/gluster1:_data



Best Regards,
Strahil Nikolov


В вторник, 22 септември 2020 г., 19:44:54 Гринуич+3, Jeremey Wise 
 написа: 






Yes.

And at one time it was fine.   I did a graceful shutdown.. and after booting it 
always seems to now have issue with the one server... of course the one hosting 
the ovirt-engine :P

# Three nodes in cluster

# Error when you hover over node


# when i select node and choose "activate"



#Gluster is working fine... this is oVirt who is confused.
[root@medusa vmstore]# mount |grep media/vmstore
medusast.penguinpages.local:/vmstore on /media/vmstore type fuse.glusterfs 
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev)
[root@medusa vmstore]# echo > /media/vmstore/test.out
[root@medusa vmstore]# ssh -f thor 'echo $HOSTNAME >> /media/vmstore/test.out'
[root@medusa vmstore]# ssh -f odin 'echo $HOSTNAME >> /media/vmstore/test.out'
[root@medusa vmstore]# ssh -f medusa 'echo $HOSTNAME >> /media/vmstore/test.out'
[root@medusa vmstore]# cat /media/vmstore/test.out

thor.penguinpages.local
odin.penguinpages.local
medusa.penguinpages.local


Ideas to fix oVirt?



On Tue, Sep 22, 2020 at 10:42 AM Strahil Nikolov  wrote:
> By the way, did you add the third host in the oVirt ?
> 
> If not , maybe that is the real problem :)
> 
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> 
> 
> В вторник, 22 септември 2020 г., 17:23:28 Гринуич+3, Jeremey Wise 
>  написа: 
> 
> 
> 
> 
> 
> Its like oVirt thinks there are only two nodes in gluster replication
> 
> 
> 
> 
> 
> # Yet it is clear the CLI shows three bricks.
> [root@medusa vms]# gluster volume status vmstore
> Status of volume: vmstore
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> --
> Brick thorst.penguinpages.local:/gluster_br
> icks/vmstore/vmstore                        49154     0          Y       9444
> Brick odinst.penguinpages.local:/gluster_br
> icks/vmstore/vmstore                        49154     0          Y       3269
> Brick medusast.penguinpages.local:/gluster_
> bricks/vmstore/vmstore                      49154     0          Y       7841
> Self-heal Daemon on localhost               N/A       N/A        Y       80152
> Self-heal Daemon on odinst.penguinpages.loc
> al                                          N/A       N/A        Y       
> 141750
> Self-heal Daemon on thorst.penguinpages.loc
> al                                          N/A       N/A        Y       
> 245870
> 
> Task Status of Volume vmstore
> --
> There are no active volume tasks
> 
> 
> 
> How do I get oVirt to re-establish reality to what Gluster sees?
> 
> 
> 
> On Tue, Sep 22, 2020 at 8:59 AM Strahil Nikolov  wrote:
>> Also in some rare cases, I have seen oVirt showing gluster as 2 out of 3 
>> bricks up , but usually it was an UI issue and you go to UI and mark a 
>> "force start" which will try to start any bricks that were down (won't 
>> affect gluster) and will wake up the UI task to verify again brick status.
>> 
>> 
>> https://github.com/gluster/gstatus is a good one to verify your cluster 
>> health , yet human's touch is priceless in any kind of technology.
>> 
>> Best Regards,
>> Strahil Nikolov
>> 
>> 
>> 
>> 
>> 
>> 
>> В вторник, 22 септември 2020 г., 15:50:35 Гринуич+3, Jeremey Wise 
>>  написа: 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> when I posted last..  in the tread I paste a roling restart.    And...  now 
>> it is replicating.
>> 
>> oVirt still showing wrong.  BUT..   I did my normal test from each of the 
>> three nodes.
>> 
>> 1) Mount Gluster file system with localhost as primary and other two as 
>> tertiary to local mount (like a client would do)
>> 2) run test file create Ex:   echo $HOSTNAME >> /media/glustervolume/test.out
>> 3) repeat from each node then read back that all are in sync.
>> 
>> I REALLY hate reboot (restart) as a fix.  I need to get better with root 
>>

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

By the way, did you add the third host in the oVirt ?

If not , maybe that is the real problem :)


Best Regards,
Strahil Nikolov






В вторник, 22 септември 2020 г., 17:23:28 Гринуич+3, Jeremey Wise 
 написа: 





Its like oVirt thinks there are only two nodes in gluster replication





# Yet it is clear the CLI shows three bricks.
[root@medusa vms]# gluster volume status vmstore
Status of volume: vmstore
Gluster process                             TCP Port  RDMA Port  Online  Pid
--
Brick thorst.penguinpages.local:/gluster_br
icks/vmstore/vmstore                        49154     0          Y       9444
Brick odinst.penguinpages.local:/gluster_br
icks/vmstore/vmstore                        49154     0          Y       3269
Brick medusast.penguinpages.local:/gluster_
bricks/vmstore/vmstore                      49154     0          Y       7841
Self-heal Daemon on localhost               N/A       N/A        Y       80152
Self-heal Daemon on odinst.penguinpages.loc
al                                          N/A       N/A        Y       141750
Self-heal Daemon on thorst.penguinpages.loc
al                                          N/A       N/A        Y       245870

Task Status of Volume vmstore
--
There are no active volume tasks



How do I get oVirt to re-establish reality to what Gluster sees?



On Tue, Sep 22, 2020 at 8:59 AM Strahil Nikolov  wrote:
> Also in some rare cases, I have seen oVirt showing gluster as 2 out of 3 
> bricks up , but usually it was an UI issue and you go to UI and mark a "force 
> start" which will try to start any bricks that were down (won't affect 
> gluster) and will wake up the UI task to verify again brick status.
> 
> 
> https://github.com/gluster/gstatus is a good one to verify your cluster 
> health , yet human's touch is priceless in any kind of technology.
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> 
> 
> В вторник, 22 септември 2020 г., 15:50:35 Гринуич+3, Jeremey Wise 
>  написа: 
> 
> 
> 
> 
> 
> 
> 
> when I posted last..  in the tread I paste a roling restart.    And...  now 
> it is replicating.
> 
> oVirt still showing wrong.  BUT..   I did my normal test from each of the 
> three nodes.
> 
> 1) Mount Gluster file system with localhost as primary and other two as 
> tertiary to local mount (like a client would do)
> 2) run test file create Ex:   echo $HOSTNAME >> /media/glustervolume/test.out
> 3) repeat from each node then read back that all are in sync.
> 
> I REALLY hate reboot (restart) as a fix.  I need to get better with root 
> cause of gluster issues if I am going to trust it.  Before when I manually 
> made the volumes and it was simply (vdo + gluster) then worst case was that 
> gluster would break... but I could always go into "brick" path and copy data 
> out.
> 
> Now with oVirt.. .and LVM and thin provisioning etc..   I am abstracted from 
> simple file recovery..  Without GLUSTER AND oVirt Engine up... all my 
> environment  and data is lost.  This means nodes moved more to "pets" then 
> cattle.
> 
> And with three nodes.. I can't afford to loose any pets. 
> 
> I will post more when I get cluster settled and work on those wierd notes 
> about quorum volumes noted on two nodes when glusterd is restarted.
> 
> Thanks,
> 
> On Tue, Sep 22, 2020 at 8:44 AM Strahil Nikolov  wrote:
>> Replication issue could mean that one of the client (FUSE mounts) is not 
>> attached to all bricks.
>> 
>> You can check the amount of clients via:
>> gluster volume status all client-list
>> 
>> 
>> As a prevention , just do a rolling restart:
>> - set a host in maintenance and mark it to stop glusterd service (I'm 
>> reffering to the UI)
>> - Activate the host , once it was moved to maintenance
>> 
>> Wait for the host's HE score to recover (silver/gold crown in UI) and then 
>> proceed with the next one.
>> 
>> Best Regards,
>> Strahil Nikolov
>> 
>> 
>> 
>> 
>> В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise 
>>  написа: 
>> 
>> 
>> 
>> 
>> 
>> 
>> I did.
>> 
>> Here are all three nodes with restart. I find it odd ... their has been a 
>> set of messages at end (see below) which I don't know enough about what 
>> oVirt laid out to know if it is bad.
>> 
>> ###
>> [root@thor vmstore]# systemctl status glusterd
>> ● glusterd.service - GlusterFS, a clustered file-system server
>>    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
>> preset: disabled)
>>   Drop-In: /etc/systemd/system/glusterd.service.d
>>            └─99-cpu.conf
>>    Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
>>      Docs: man:glusterd(8)
>>   Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
>> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>>  Main PID: 2113 (glusterd)
>>     Tasks: 151 (limit: 1235410)
>>    Memory:

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

That's really wierd.
I would give the engine a 'Windows'-style fix (a.k.a. reboot).

I guess some of the engine's internal processes crashed/looped and it doesn't 
see the reality.

Best Regards,
Strahil Nikolov






В вторник, 22 септември 2020 г., 16:27:25 Гринуич+3, Jeremey Wise 
 написа: 





Its like oVirt thinks there are only two nodes in gluster replication





# Yet it is clear the CLI shows three bricks.
[root@medusa vms]# gluster volume status vmstore
Status of volume: vmstore
Gluster process                             TCP Port  RDMA Port  Online  Pid
--
Brick thorst.penguinpages.local:/gluster_br
icks/vmstore/vmstore                        49154     0          Y       9444
Brick odinst.penguinpages.local:/gluster_br
icks/vmstore/vmstore                        49154     0          Y       3269
Brick medusast.penguinpages.local:/gluster_
bricks/vmstore/vmstore                      49154     0          Y       7841
Self-heal Daemon on localhost               N/A       N/A        Y       80152
Self-heal Daemon on odinst.penguinpages.loc
al                                          N/A       N/A        Y       141750
Self-heal Daemon on thorst.penguinpages.loc
al                                          N/A       N/A        Y       245870

Task Status of Volume vmstore
--
There are no active volume tasks



How do I get oVirt to re-establish reality to what Gluster sees?



On Tue, Sep 22, 2020 at 8:59 AM Strahil Nikolov  wrote:
> Also in some rare cases, I have seen oVirt showing gluster as 2 out of 3 
> bricks up , but usually it was an UI issue and you go to UI and mark a "force 
> start" which will try to start any bricks that were down (won't affect 
> gluster) and will wake up the UI task to verify again brick status.
> 
> 
> https://github.com/gluster/gstatus is a good one to verify your cluster 
> health , yet human's touch is priceless in any kind of technology.
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> 
> 
> В вторник, 22 септември 2020 г., 15:50:35 Гринуич+3, Jeremey Wise 
>  написа: 
> 
> 
> 
> 
> 
> 
> 
> when I posted last..  in the tread I paste a roling restart.    And...  now 
> it is replicating.
> 
> oVirt still showing wrong.  BUT..   I did my normal test from each of the 
> three nodes.
> 
> 1) Mount Gluster file system with localhost as primary and other two as 
> tertiary to local mount (like a client would do)
> 2) run test file create Ex:   echo $HOSTNAME >> /media/glustervolume/test.out
> 3) repeat from each node then read back that all are in sync.
> 
> I REALLY hate reboot (restart) as a fix.  I need to get better with root 
> cause of gluster issues if I am going to trust it.  Before when I manually 
> made the volumes and it was simply (vdo + gluster) then worst case was that 
> gluster would break... but I could always go into "brick" path and copy data 
> out.
> 
> Now with oVirt.. .and LVM and thin provisioning etc..   I am abstracted from 
> simple file recovery..  Without GLUSTER AND oVirt Engine up... all my 
> environment  and data is lost.  This means nodes moved more to "pets" then 
> cattle.
> 
> And with three nodes.. I can't afford to loose any pets. 
> 
> I will post more when I get cluster settled and work on those wierd notes 
> about quorum volumes noted on two nodes when glusterd is restarted.
> 
> Thanks,
> 
> On Tue, Sep 22, 2020 at 8:44 AM Strahil Nikolov  wrote:
>> Replication issue could mean that one of the client (FUSE mounts) is not 
>> attached to all bricks.
>> 
>> You can check the amount of clients via:
>> gluster volume status all client-list
>> 
>> 
>> As a prevention , just do a rolling restart:
>> - set a host in maintenance and mark it to stop glusterd service (I'm 
>> reffering to the UI)
>> - Activate the host , once it was moved to maintenance
>> 
>> Wait for the host's HE score to recover (silver/gold crown in UI) and then 
>> proceed with the next one.
>> 
>> Best Regards,
>> Strahil Nikolov
>> 
>> 
>> 
>> 
>> В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise 
>>  написа: 
>> 
>> 
>> 
>> 
>> 
>> 
>> I did.
>> 
>> Here are all three nodes with restart. I find it odd ... their has been a 
>> set of messages at end (see below) which I don't know enough about what 
>> oVirt laid out to know if it is bad.
>> 
>> ###
>> [root@thor vmstore]# systemctl status glusterd
>> ● glusterd.service - GlusterFS, a clustered file-system server
>>    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
>> preset: disabled)
>>   Drop-In: /etc/systemd/system/glusterd.service.d
>>            └─99-cpu.conf
>>    Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
>>      Docs: man:glusterd(8)
>>   Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
>> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited,

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

Also in some rare cases, I have seen oVirt showing gluster as 2 out of 3 bricks 
up , but usually it was an UI issue and you go to UI and mark a "force start" 
which will try to start any bricks that were down (won't affect gluster) and 
will wake up the UI task to verify again brick status.

https://github.com/gluster/gstatus is a good one to verify your cluster health 
, yet human's touch is priceless in any kind of technology.

Best Regards,
Strahil Nikolov

В вторник, 22 септември 2020 г., 15:50:35 Гринуич+3, Jeremey Wise 
 написа: 

when I posted last..  in the tread I paste a roling restart.    And...  now it 
is replicating.

oVirt still showing wrong.  BUT..   I did my normal test from each of the three 
nodes.

1) Mount Gluster file system with localhost as primary and other two as 
tertiary to local mount (like a client would do)
2) run test file create Ex:   echo $HOSTNAME >> /media/glustervolume/test.out
3) repeat from each node then read back that all are in sync.

I REALLY hate reboot (restart) as a fix.  I need to get better with root cause 
of gluster issues if I am going to trust it.  Before when I manually made the 
volumes and it was simply (vdo + gluster) then worst case was that gluster 
would break... but I could always go into "brick" path and copy data out.

Now with oVirt.. .and LVM and thin provisioning etc..   I am abstracted from 
simple file recovery..  Without GLUSTER AND oVirt Engine up... all my 
environment  and data is lost.  This means nodes moved more to "pets" then 
cattle.

And with three nodes.. I can't afford to loose any pets. 

I will post more when I get cluster settled and work on those wierd notes about 
quorum volumes noted on two nodes when glusterd is restarted.

Thanks,

On Tue, Sep 22, 2020 at 8:44 AM Strahil Nikolov  wrote:
> Replication issue could mean that one of the client (FUSE mounts) is not 
> attached to all bricks.
> 
> You can check the amount of clients via:
> gluster volume status all client-list
> 
> 
> As a prevention , just do a rolling restart:
> - set a host in maintenance and mark it to stop glusterd service (I'm 
> reffering to the UI)
> - Activate the host , once it was moved to maintenance
> 
> Wait for the host's HE score to recover (silver/gold crown in UI) and then 
> proceed with the next one.
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise 
>  написа: 
> 
> 
> 
> 
> 
> 
> I did.
> 
> Here are all three nodes with restart. I find it odd ... their has been a set 
> of messages at end (see below) which I don't know enough about what oVirt 
> laid out to know if it is bad.
> 
> ###
> [root@thor vmstore]# systemctl status glusterd
> ● glusterd.service - GlusterFS, a clustered file-system server
>    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
> preset: disabled)
>   Drop-In: /etc/systemd/system/glusterd.service.d
>            └─99-cpu.conf
>    Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
>      Docs: man:glusterd(8)
>   Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>  Main PID: 2113 (glusterd)
>     Tasks: 151 (limit: 1235410)
>    Memory: 3.8G
>       CPU: 6min 46.050s
>    CGroup: /glusterfs.slice/glusterd.service
>            ├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level 
> INFO
>            ├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data -p 
> /var/run/gluster/shd/data/data-shd.pid -l /var/log/glusterfs/glustershd.log 
> -S /var/run/gluster/2f41374c2e36bf4d.socket --xlator-option 
> *replicate*.node-uu>
>            ├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p 
> /var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
>  -S /var/r>
>            ├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine -p 
> /var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
>            ├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore 
> -p 
> /var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
>            └─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p 
> /var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid
>  -S /var/run/glu>
> 
> Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a 
> clustered file-system server...
> Sep 21 20:32:26 thor.penguinpages.local systemd[1]: Started GlusterFS, a 
> clustered file-system server.
> Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
>

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

Usually I first start with:
'gluster volume heal  info summary'

Anything that is not 'Connected' is bad.

Yeah, the abstraction is not so nice, but the good thing is that you can always 
extract the data from a single node left (it will require to play a little bit 
with the quorum of the volume).

Usually I have seen that the FUSE fails to reconnect to a "gone bad and 
recovered" brick and then you got that endless healing (as FUSE will write the 
data to only 2 out of 3 bricks and then a heal is pending :D ).

I would go with the gluster logs and the brick logs and then you can dig deeper 
if you suspect network issue.

Best Regards,
Strahil Nikolov

В вторник, 22 септември 2020 г., 15:50:35 Гринуич+3, Jeremey Wise 
 написа: 

when I posted last..  in the tread I paste a roling restart.    And...  now it 
is replicating.

oVirt still showing wrong.  BUT..   I did my normal test from each of the three 
nodes.

1) Mount Gluster file system with localhost as primary and other two as 
tertiary to local mount (like a client would do)
2) run test file create Ex:   echo $HOSTNAME >> /media/glustervolume/test.out
3) repeat from each node then read back that all are in sync.

I REALLY hate reboot (restart) as a fix.  I need to get better with root cause 
of gluster issues if I am going to trust it.  Before when I manually made the 
volumes and it was simply (vdo + gluster) then worst case was that gluster 
would break... but I could always go into "brick" path and copy data out.

Now with oVirt.. .and LVM and thin provisioning etc..   I am abstracted from 
simple file recovery..  Without GLUSTER AND oVirt Engine up... all my 
environment  and data is lost.  This means nodes moved more to "pets" then 
cattle.

And with three nodes.. I can't afford to loose any pets. 

I will post more when I get cluster settled and work on those wierd notes about 
quorum volumes noted on two nodes when glusterd is restarted.

Thanks,

On Tue, Sep 22, 2020 at 8:44 AM Strahil Nikolov  wrote:
> Replication issue could mean that one of the client (FUSE mounts) is not 
> attached to all bricks.
> 
> You can check the amount of clients via:
> gluster volume status all client-list
> 
> 
> As a prevention , just do a rolling restart:
> - set a host in maintenance and mark it to stop glusterd service (I'm 
> reffering to the UI)
> - Activate the host , once it was moved to maintenance
> 
> Wait for the host's HE score to recover (silver/gold crown in UI) and then 
> proceed with the next one.
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise 
>  написа: 
> 
> 
> 
> 
> 
> 
> I did.
> 
> Here are all three nodes with restart. I find it odd ... their has been a set 
> of messages at end (see below) which I don't know enough about what oVirt 
> laid out to know if it is bad.
> 
> ###
> [root@thor vmstore]# systemctl status glusterd
> ● glusterd.service - GlusterFS, a clustered file-system server
>    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
> preset: disabled)
>   Drop-In: /etc/systemd/system/glusterd.service.d
>            └─99-cpu.conf
>    Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
>      Docs: man:glusterd(8)
>   Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>  Main PID: 2113 (glusterd)
>     Tasks: 151 (limit: 1235410)
>    Memory: 3.8G
>       CPU: 6min 46.050s
>    CGroup: /glusterfs.slice/glusterd.service
>            ├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level 
> INFO
>            ├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data -p 
> /var/run/gluster/shd/data/data-shd.pid -l /var/log/glusterfs/glustershd.log 
> -S /var/run/gluster/2f41374c2e36bf4d.socket --xlator-option 
> *replicate*.node-uu>
>            ├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p 
> /var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
>  -S /var/r>
>            ├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine -p 
> /var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
>            ├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore 
> -p 
> /var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
>            └─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
> --volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p 
> /var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid
>  -S /var/run/glu>
> 
> Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a 
> clustered file-system server...

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-22 Thread Jeremey Wise

when I posted last..  in the tread I paste a roling restart.And...  now
it is replicating.

oVirt still showing wrong.  BUT..   I did my normal test from each of the
three nodes.

1) Mount Gluster file system with localhost as primary and other two as
tertiary to local mount (like a client would do)
2) run test file create Ex:   echo $HOSTNAME >>
/media/glustervolume/test.out
3) repeat from each node then read back that all are in sync.

I REALLY hate reboot (restart) as a fix.  I need to get better with root
cause of gluster issues if I am going to trust it.  Before when I manually
made the volumes and it was simply (vdo + gluster) then worst case was that
gluster would break... but I could always go into "brick" path and copy
data out.

Now with oVirt.. .and LVM and thin provisioning etc..   I am abstracted
from simple file recovery..  Without GLUSTER AND oVirt Engine up... all my
environment  and data is lost.  This means nodes moved more to "pets" then
cattle.

And with three nodes.. I can't afford to loose any pets.

I will post more when I get cluster settled and work on those wierd notes
about quorum volumes noted on two nodes when glusterd is restarted.

Thanks,

On Tue, Sep 22, 2020 at 8:44 AM Strahil Nikolov 
wrote:

> Replication issue could mean that one of the client (FUSE mounts) is not
> attached to all bricks.
>
> You can check the amount of clients via:
> gluster volume status all client-list
>
>
> As a prevention , just do a rolling restart:
> - set a host in maintenance and mark it to stop glusterd service (I'm
> reffering to the UI)
> - Activate the host , once it was moved to maintenance
>
> Wait for the host's HE score to recover (silver/gold crown in UI) and then
> proceed with the next one.
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
> В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise <
> jeremey.w...@gmail.com> написа:
>
>
>
>
>
>
> I did.
>
> Here are all three nodes with restart. I find it odd ... their has been a
> set of messages at end (see below) which I don't know enough about what
> oVirt laid out to know if it is bad.
>
> ###
> [root@thor vmstore]# systemctl status glusterd
> ● glusterd.service - GlusterFS, a clustered file-system server
>Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled;
> vendor preset: disabled)
>   Drop-In: /etc/systemd/system/glusterd.service.d
>└─99-cpu.conf
>Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
>  Docs: man:glusterd(8)
>   Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid
> --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
>  Main PID: 2113 (glusterd)
> Tasks: 151 (limit: 1235410)
>Memory: 3.8G
>   CPU: 6min 46.050s
>CGroup: /glusterfs.slice/glusterd.service
>├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
> INFO
>├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data
> -p /var/run/gluster/shd/data/data-shd.pid -l
> /var/log/glusterfs/glustershd.log -S
> /var/run/gluster/2f41374c2e36bf4d.socket --xlator-option
> *replicate*.node-uu>
>├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local
> --volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p
> /var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
> -S /var/r>
>├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local
> --volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine
> -p
> /var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
>├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local
> --volfile-id
> vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore -p
> /var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
>└─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local
> --volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p
> /var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid
> -S /var/run/glu>
>
> Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a
> clustered file-system server...
> Sep 21 20:32:26 thor.penguinpages.local systemd[1]: Started GlusterFS, a
> clustered file-system server.
> Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
> 00:32:28.605674] C [MSGID: 106003]
> [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
> 0-management: Server quorum regained for volume data. Starting lo>
> Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
> 00:32:28.639490] C [MSGID: 106003]
> [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
> 0-management: Server quorum regained for volume engine. Starting >
> Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
> 00:32:28.680665] C [MSGID: 106003]
> [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
>

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

At around Sep 21 20:33 local time , you got  a loss of quorum - that's not good.

Could it be a network 'hicup' ?

Best Regards,
Strahil Nikolov






В вторник, 22 септември 2020 г., 15:05:16 Гринуич+3, Jeremey Wise 
 написа: 






I did.

Here are all three nodes with restart. I find it odd ... their has been a set 
of messages at end (see below) which I don't know enough about what oVirt laid 
out to know if it is bad.

###
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
preset: disabled)
  Drop-In: /etc/systemd/system/glusterd.service.d
           └─99-cpu.conf
   Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
     Docs: man:glusterd(8)
  Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
--log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2113 (glusterd)
    Tasks: 151 (limit: 1235410)
   Memory: 3.8G
      CPU: 6min 46.050s
   CGroup: /glusterfs.slice/glusterd.service
           ├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
           ├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data -p 
/var/run/gluster/shd/data/data-shd.pid -l /var/log/glusterfs/glustershd.log -S 
/var/run/gluster/2f41374c2e36bf4d.socket --xlator-option *replicate*.node-uu>
           ├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p 
/var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
 -S /var/r>
           ├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine -p 
/var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
           ├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore 
-p 
/var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
           └─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p 
/var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid 
-S /var/run/glu>

Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a 
clustered file-system server...
Sep 21 20:32:26 thor.penguinpages.local systemd[1]: Started GlusterFS, a 
clustered file-system server.
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.605674] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume data. Starting lo>
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.639490] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume engine. Starting >
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.680665] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume vmstore. Starting>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.813409] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
0-data-client-0: server 172.16.101.101:24007 has not responded in the last 30 
seconds, discon>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.815147] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
2-engine-client-0: server 172.16.101.101:24007 has not responded in the last 30 
seconds, disc>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.818735] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
4-vmstore-client-0: server 172.16.101.101:24007 has not responded in the last 
30 seconds, dis>
Sep 21 20:33:36 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:36.816978] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
3-iso-client-0: server 172.16.101.101:24007 has not responded in the last 42 
seconds, disconn>
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]# systemctl restart glusterd
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
preset: disabled)
  Drop-In: /etc/systemd/system/glusterd.service.d
           └─99-cpu.conf
   Active: active (running) since Tue 2020-09-22 07:24:34 EDT; 2s ago
     Docs: man:glusterd(8)
  Process: 245831 ExecStart=/usr/sbin/glusterd -p

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

Replication issue could mean that one of the client (FUSE mounts) is not 
attached to all bricks.

You can check the amount of clients via:
gluster volume status all client-list


As a prevention , just do a rolling restart:
- set a host in maintenance and mark it to stop glusterd service (I'm reffering 
to the UI)
- Activate the host , once it was moved to maintenance

Wait for the host's HE score to recover (silver/gold crown in UI) and then 
proceed with the next one.

Best Regards,
Strahil Nikolov




В вторник, 22 септември 2020 г., 14:55:35 Гринуич+3, Jeremey Wise 
 написа: 






I did.

Here are all three nodes with restart. I find it odd ... their has been a set 
of messages at end (see below) which I don't know enough about what oVirt laid 
out to know if it is bad.

###
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor 
preset: disabled)
  Drop-In: /etc/systemd/system/glusterd.service.d
           └─99-cpu.conf
   Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
     Docs: man:glusterd(8)
  Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid 
--log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2113 (glusterd)
    Tasks: 151 (limit: 1235410)
   Memory: 3.8G
      CPU: 6min 46.050s
   CGroup: /glusterfs.slice/glusterd.service
           ├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
           ├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data -p 
/var/run/gluster/shd/data/data-shd.pid -l /var/log/glusterfs/glustershd.log -S 
/var/run/gluster/2f41374c2e36bf4d.socket --xlator-option *replicate*.node-uu>
           ├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p 
/var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
 -S /var/r>
           ├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine -p 
/var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
           ├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore 
-p 
/var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
           └─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local 
--volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p 
/var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid 
-S /var/run/glu>

Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a 
clustered file-system server...
Sep 21 20:32:26 thor.penguinpages.local systemd[1]: Started GlusterFS, a 
clustered file-system server.
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.605674] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume data. Starting lo>
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.639490] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume engine. Starting >
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22 
00:32:28.680665] C [MSGID: 106003] 
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: 
Server quorum regained for volume vmstore. Starting>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.813409] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
0-data-client-0: server 172.16.101.101:24007 has not responded in the last 30 
seconds, discon>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.815147] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
2-engine-client-0: server 172.16.101.101:24007 has not responded in the last 30 
seconds, disc>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:24.818735] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
4-vmstore-client-0: server 172.16.101.101:24007 has not responded in the last 
30 seconds, dis>
Sep 21 20:33:36 thor.penguinpages.local glustershd[2914]: [2020-09-22 
00:33:36.816978] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 
3-iso-client-0: server 172.16.101.101:24007 has not responded in the last 42 
seconds, disconn>
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]# systemctl restart glusterd
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active

2020-09-22 Thread Jeremey Wise

I did.

Here are all three nodes with restart. I find it odd ... their has been a
set of messages at end (see below) which I don't know enough about what
oVirt laid out to know if it is bad.

###
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled;
vendor preset: disabled)
  Drop-In: /etc/systemd/system/glusterd.service.d
   └─99-cpu.conf
   Active: active (running) since Mon 2020-09-21 20:32:26 EDT; 10h ago
 Docs: man:glusterd(8)
  Process: 2001 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid
--log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2113 (glusterd)
Tasks: 151 (limit: 1235410)
   Memory: 3.8G
  CPU: 6min 46.050s
   CGroup: /glusterfs.slice/glusterd.service
   ├─ 2113 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
INFO
   ├─ 2914 /usr/sbin/glusterfs -s localhost --volfile-id shd/data
-p /var/run/gluster/shd/data/data-shd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/gluster/2f41374c2e36bf4d.socket --xlator-option
*replicate*.node-uu>
   ├─ 9342 /usr/sbin/glusterfsd -s thorst.penguinpages.local
--volfile-id data.thorst.penguinpages.local.gluster_bricks-data-data -p
/var/run/gluster/vols/data/thorst.penguinpages.local-gluster_bricks-data-data.pid
-S /var/r>
   ├─ 9433 /usr/sbin/glusterfsd -s thorst.penguinpages.local
--volfile-id engine.thorst.penguinpages.local.gluster_bricks-engine-engine
-p
/var/run/gluster/vols/engine/thorst.penguinpages.local-gluster_bricks-engine-engine.p>
   ├─ 9444 /usr/sbin/glusterfsd -s thorst.penguinpages.local
--volfile-id
vmstore.thorst.penguinpages.local.gluster_bricks-vmstore-vmstore -p
/var/run/gluster/vols/vmstore/thorst.penguinpages.local-gluster_bricks-vmstore-vms>
   └─35639 /usr/sbin/glusterfsd -s thorst.penguinpages.local
--volfile-id iso.thorst.penguinpages.local.gluster_bricks-iso-iso -p
/var/run/gluster/vols/iso/thorst.penguinpages.local-gluster_bricks-iso-iso.pid
-S /var/run/glu>

Sep 21 20:32:24 thor.penguinpages.local systemd[1]: Starting GlusterFS, a
clustered file-system server...
Sep 21 20:32:26 thor.penguinpages.local systemd[1]: Started GlusterFS, a
clustered file-system server.
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
00:32:28.605674] C [MSGID: 106003]
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
0-management: Server quorum regained for volume data. Starting lo>
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
00:32:28.639490] C [MSGID: 106003]
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
0-management: Server quorum regained for volume engine. Starting >
Sep 21 20:32:28 thor.penguinpages.local glusterd[2113]: [2020-09-22
00:32:28.680665] C [MSGID: 106003]
[glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action]
0-management: Server quorum regained for volume vmstore. Starting>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22
00:33:24.813409] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired]
0-data-client-0: server 172.16.101.101:24007 has not responded in the last
30 seconds, discon>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22
00:33:24.815147] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired]
2-engine-client-0: server 172.16.101.101:24007 has not responded in the
last 30 seconds, disc>
Sep 21 20:33:24 thor.penguinpages.local glustershd[2914]: [2020-09-22
00:33:24.818735] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired]
4-vmstore-client-0: server 172.16.101.101:24007 has not responded in the
last 30 seconds, dis>
Sep 21 20:33:36 thor.penguinpages.local glustershd[2914]: [2020-09-22
00:33:36.816978] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired]
3-iso-client-0: server 172.16.101.101:24007 has not responded in the last
42 seconds, disconn>
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]#
[root@thor vmstore]# systemctl restart glusterd
[root@thor vmstore]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled;
vendor preset: disabled)
  Drop-In: /etc/systemd/system/glusterd.service.d
   └─99-cpu.conf
   Active: active (running) since Tue 2020-09-22 07:24:34 EDT; 2s ago
 Docs: man:glusterd(8)
  Process: 245831 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid
--log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 245832 (glusterd)
Tasks: 151 (limit: 1235410)
   Memory: 3.8G
  CPU: 132ms
   CGroup: /glusterfs.slice/glusterd.service
   ├─  2914 /usr/sbin/glusterfs -s localhost

[ovirt-users] Re: oVirt - Gluster Node Offline but Bricks Active