[ovirt-users] Re: hosted engine storage does not heal

2020-02-13 Thread George Vasilopoulos

it seems that this exactly was the problem

Thank you so much!

Στις 13/2/20 5:33 μ.μ., ο Strahil Nikolov έγραψε:

On February 13, 2020 5:20:18 PM GMT+02:00, g.vasilopou...@uoc.gr wrote:

Hello
We have a problem with hosted engine storage after updating one host
which serves as a gluster server for the engine (the setup is gluster
replica 3 with local disks from 3 hypervisors)
Volume heal command shows
[root@o5-car0118 engine]# gluster volume heal engine info
Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta

/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta

Status: Connected
Number of entries: 2

Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine
/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta

/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta

Status: Connected
Number of entries: 2

Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0

On all the gluster servers
I notice that the affected directories have date in 1970.
[root@o5-car0118 images]# ls -al
σύνολο 24
drwxr-xr-x. 23 vdsm kvm 8192 Σεπ  24 12:07 .
drwxr-xr-x.  6 vdsm kvm   64 Σεπ  19  2018 ..
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
2bac658f-70ce-4adb-ab68-a0f0c205c70c
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
3034a69c-b5b5-46fa-a393-59ea46635142
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
66dbce25-8863-42b5-904a-484f8e9c225a
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
6c049108-28f7-47d9-8d54-4ac2697dcba8
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
72702607-1896-420d-931a-42c9f01d37f1
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
7c617da4-ab6b-4791-80be-541f5be60dd8
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
902a16d3-6494-4840-a528-b49972f9c332
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
96fd6116-7983-4385-bca6-e6ca8edc94ca
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
abd875cd-96b6-47a6-b6a3-ae35300a21cc
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
add7bc92-1a40-474d-9255-53ac861b75ed
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
b7b06df7-465f-4fc7-a214-033b7dca6bc7
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
c0ecacac-26c6-40d9-87da-af17d9de8d21
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
c4d2d5da-2a15-4735-8919-324ae8372064
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
c7e0c784-bb8e-4024-95df-b6f4267b0208
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
d1f1ff5a-387d-442c-9240-1c58e4d6f8a7
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
d3e172cb-b6dd-4867-a9cd-f4fa006648bc
drwxr-xr-x.  2 vdsm kvm 8192 Ιαν   1  1970
e3a3ef50-56b6-48b0-a9f8-2d6382e2286e  <-
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
e477ec02-11ab-4d92-b5fd-44e91fbde7f9
drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
e839485b-b0be-47f6-9847-b691e02ce9a4
drwxr-xr-x.  2 vdsm kvm 8192 Ιαν   1  1970
f5e576d4-eea7-431b-a0f0-f8a557006471 <-

I think this has something to do with a gluster bug.
Is there a way to correct this and heal the volume?
Thank you!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHC5TPN6DU5ZV7UY6ZYB4YMEVE6JO3SY/

For the meta files - check on the bricks  which is newest. Usually only the 
timestamp inside  is different.

If data is the same -> rsync the newest file from the brick to the other bricks and 
use 'gluster volume heal  full' (for me it  never worked  without 'full').

We are expecting fix for the meta files  with different gfid very soon.


For the timestamp - write an e-mail to the gluster-users mail list, as they can 
help better.

Best Regards,
Strahil Nikolov


--
Βασιλόπουλος Γιώργος
Ηλεκτρολόγος Μηχανικός Τ.Ε.
Διαχειριστής Υπολ. Συστημάτων

Πανεπιστήμιο Κρήτης
Κ.Υ.Υ.Τ.Π.Ε.
Τμήμα Επικοινωνιών και Δικτύων
Βούτες Ηρακλείου 70013
Τηλ   : 2810393310
email : g.vasilopou...@uoc.gr
http://www.ucnet.uoc.gr
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/E4NI3QNGY7S32IVJ4LURHYHSKV5RP6ZM/


[ovirt-users] Re: hosted engine storage does not heal

2020-02-13 Thread Strahil Nikolov
On February 13, 2020 5:20:18 PM GMT+02:00, g.vasilopou...@uoc.gr wrote:
>Hello 
>We have a problem with hosted engine storage after updating one host
>which serves as a gluster server for the engine (the setup is gluster
>replica 3 with local disks from 3 hypervisors)
>Volume heal command shows 
>[root@o5-car0118 engine]# gluster volume heal engine info
>Brick o5-car0118.gfs-int.uoc.gr:/gluster_bricks/engine/engine
>/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
>
>/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
>
>Status: Connected
>Number of entries: 2
>
>Brick o2-car0121.gfs-int.uoc.gr:/gluster_bricks/engine/engine
>/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/e3a3ef50-56b6-48b0-a9f8-2d6382e2286e/b6973d5b-cc7a-4abd-8d2d-94f551936a97.meta
>
>/a1ecca6f-1487-492d-b9bf-d3fb60f1bc99/images/f5e576d4-eea7-431b-a0f0-f8a557006471/464518c5-f79c-4271-9d97-995981cde2cb.meta
>
>Status: Connected
>Number of entries: 2
>
>Brick o9-car0114.gfs-int.uoc.gr:/gluster_bricks/engine/engine
>Status: Connected
>Number of entries: 0
>
>On all the gluster servers 
>I notice that the affected directories have date in 1970. 
>[root@o5-car0118 images]# ls -al
>σύνολο 24
>drwxr-xr-x. 23 vdsm kvm 8192 Σεπ  24 12:07 .
>drwxr-xr-x.  6 vdsm kvm   64 Σεπ  19  2018 ..
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>2bac658f-70ce-4adb-ab68-a0f0c205c70c
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>3034a69c-b5b5-46fa-a393-59ea46635142
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>5538ae6b-ccc6-4861-b71b-6b2c7af2e0ab
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>66dbce25-8863-42b5-904a-484f8e9c225a
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>6c049108-28f7-47d9-8d54-4ac2697dcba8
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>72702607-1896-420d-931a-42c9f01d37f1
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>7c617da4-ab6b-4791-80be-541f5be60dd8
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>902a16d3-6494-4840-a528-b49972f9c332
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>96fd6116-7983-4385-bca6-e6ca8edc94ca
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>abd875cd-96b6-47a6-b6a3-ae35300a21cc
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>add7bc92-1a40-474d-9255-53ac861b75ed
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>b7b06df7-465f-4fc7-a214-033b7dca6bc7
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>c0ecacac-26c6-40d9-87da-af17d9de8d21
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>c4d2d5da-2a15-4735-8919-324ae8372064
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>c7e0c784-bb8e-4024-95df-b6f4267b0208
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>d1f1ff5a-387d-442c-9240-1c58e4d6f8a7
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>d3e172cb-b6dd-4867-a9cd-f4fa006648bc
>drwxr-xr-x.  2 vdsm kvm 8192 Ιαν   1  1970
>e3a3ef50-56b6-48b0-a9f8-2d6382e2286e  <-
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>e477ec02-11ab-4d92-b5fd-44e91fbde7f9
>drwxr-xr-x.  2 vdsm kvm  149 Αύγ   2  2019
>e839485b-b0be-47f6-9847-b691e02ce9a4
>drwxr-xr-x.  2 vdsm kvm 8192 Ιαν   1  1970
>f5e576d4-eea7-431b-a0f0-f8a557006471 <-
>
>I think this has something to do with a gluster bug. 
>Is there a way to correct this and heal the volume?
>Thank you!
>___
>Users mailing list -- users@ovirt.org
>To unsubscribe send an email to users-le...@ovirt.org
>Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/EHC5TPN6DU5ZV7UY6ZYB4YMEVE6JO3SY/

For the meta files - check on the bricks  which is newest. Usually only the 
timestamp inside  is different.

If data is the same -> rsync the newest file from the brick to the other bricks 
and use 'gluster volume heal  full' (for me it  never worked  without 
'full').

We are expecting fix for the meta files  with different gfid very soon.


For the timestamp - write an e-mail to the gluster-users mail list, as they can 
help better.

Best Regards,
Strahil Nikolov
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FUVBJF5TJLF542V7VYHRGDCNUN6VGWYG/