Hello,

Many thanks for your email.
I should add that this is a test environment we set up in preparation for a
planned CentOS 7 / oVirt 4.3 upgrade to CentOS 8 Streams / oVirt 4.5
upgrade in one of our old(er) oVirt clusters.
In this case, we blew up the software RAID during the OS replacement
(CentOS 7 -> 8) so have a host, but no storage.
And as an added bonus, the FS locations are a bit different. (due MD
changes we made during the blowup).

So, essentially the host is alive, but we need to create a new brick using
a known good brick.
A couple of questions:
Assuming I have a known good brick to copy but the FS location is different
and given the fact I cannot simply remove/add brick, how do I change the
brick path?
Old location:
office-wx-hv1-lab-gfs:/mnt/LogGFSData/brick
New location:
office-wx-hv1-lab-gfs.localdomain:/gluster/brick/data/brick
Thanks again,
Gilboa

On Mon, Jul 18, 2022 at 1:32 AM Patrick Hibbs <hibbsncc1...@gmail.com>
wrote:

> What you are missing is the fact that gluster requires more than one set
> of bricks to recover from a dead host. I.e. In your set up, you'd need 6
> hosts. 4x replicas and 2x arbiters with at least one set (2x replicas and
> 1x arbiter) operational bare minimum.
> Automated commands to fix the volume do not exist otherwise. (It's a
> Gluster limitation.) This can be fixed manually however.
>
> Standard Disclaimer: Back up your data first! Fixing this issue requires
> manual intervention. Reader assumes all responsiblity for any action
> resulting from the instructions below. Etc.
>
> If it's just a dead brick, (i.e. the host is still functional), all you
> really need to do is replace the underlying storage:
>
> 1. Take the gluster volume offline.
> 2. Remove the bad storage device, and attach the replacement.
> 3. rsync / scp / etc. the data from a known good brick (be sure to include
> hidden files / preserve file times and ownership / SELinux labels / etc. ).
> 4. Restart the gluster volume.
>
> Gluster *might* still need to heal everything after all of that, but it
> should start the volume and get it running again.
>
> If the host itself is dead, (and the underlying storage is still
> functional), you can just move the underlying storage over to the new host:
>
> 1. Take the gluster volume offline.
> 2. Attach the old storage.
> 3. Fix up the ids on the volume file. (
> https://serverfault.com/questions/631365/rename-a-glusterfs-peer)
> 4. Restart the gluster volume.
>
> If both the host and underlying storage are dead, you'll need to do both
> tasks:
>
> 1. Take the gluster volume offline.
> 2. Attach the new storage.
> 3. rsync / scp / etc. the data from a known good brick (be sure to
> include hidden files / preserve file times and ownership / SELinux labels /
> etc. ).
> 4. Fix up the ids on the volume file.
> 5. Restart the gluster volume.
>
> Keep in mind one thing however: If the gluster host you are replacing is
> used by oVirt to connect to the volume (I.e. It's the host named in the
> volume config in the Admin portal). The new host will need to retain the
> old hostname / IP, or you'll need to update oVirt's config. Otherwise the
> VM hosts will wind up in Unassigned / Non-functional status.
>
> - Patrick Hibbs
>
> On Sun, 2022-07-17 at 22:15 +0300, Gilboa Davara wrote:
>
> Hello all,
>
> I'm attempting to replace a dead host in a replica 2 + arbiter gluster
> setup and replace it with a new host.
> I've already set up a new host (same hostname..localdomain) and got into
> the cluster.
>
> $ gluster peer status
> Number of Peers: 2
>
> Hostname: office-wx-hv3-lab-gfs
> Uuid: 4e13f796-b818-4e07-8523-d84eb0faa4f9
> State: Peer in Cluster (Connected)
>
> Hostname: office-wx-hv1-lab-gfs.localdomain <------ This is a new host.
> Uuid: eee17c74-0d93-4f92-b81d-87f6b9c2204d
> State: Peer in Cluster (Connected)
>
> $ gluster volume info GV2Data
>  Volume Name: GV2Data
> Type: Replicate
> Volume ID: c1946fc2-ed94-4b9f-9da3-f0f1ee90f303
> Status: Stopped
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: office-wx-hv1-lab-gfs:/mnt/LogGFSData/brick  <------ This is the
> dead host.
> Brick2: office-wx-hv2-lab-gfs:/mnt/LogGFSData/brick
> Brick3: office-wx-hv3-lab-gfs:/mnt/LogGFSData/brick (arbiter)
> ...
>
> Looking at the docs, it seems that I need to remove the dead brick.
>
> $ gluster volume remove-brick GV2Data
> office-wx-hv1-lab-gfs:/mnt/LogGFSData/brick start
> Running remove-brick with cluster.force-migration enabled can result in
> data corruption. It is safer to disable this option so that files that
> receive writes during migration are not migrated.
> Files that are not migrated can then be manually copied after the
> remove-brick commit operation.
> Do you want to continue with your current cluster.force-migration
> settings? (y/n) y
> volume remove-brick start: failed: Removing bricks from replicate
> configuration is not allowed without reducing replica count explicitly
>
> So I guess I need to drop from replica 2 + arbiter to replica 1 + arbiter
> (?).
>
> $ gluster volume remove-brick GV2Data replica 1
> office-wx-hv1-lab-gfs:/mnt/LogGFSData/brick start
> Running remove-brick with cluster.force-migration enabled can result in
> data corruption. It is safer to disable this option so that files that
> receive writes during migration are not migrated.
> Files that are not migrated can then be manually copied after the
> remove-brick commit operation.
> Do you want to continue with your current cluster.force-migration
> settings? (y/n) y
> volume remove-brick start: failed: need 2(xN) bricks for reducing replica
> count of the volume from 3 to 1
>
> ... What am I missing?
>
> - Gilboa
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/OIXTFTJREUAHGP3WUW7DFL3VJNEMFJLF/
>
>
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/MAEPVKYSBBY3ZXO7T3VHVCPLQINDLGFO/
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H776LCZH56DBD4ZCIMCECZM2S5CAPQRO/

Reply via email to