Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?
If I were you I follow the following steps. Stop the rebalance and fix the cluster health first. Bring up the down server, replace server4:brick4 with a new disk, format and be sure it is started, then start a full heal. Without all bricks up full heal will not start. The you can continue with rebalance. On Fri, Feb 2, 2018 at 1:27 PM, Alessandro Ipe wrote: > Hi, > > > I simplified the config in my first email, but I actually have 2x4 servers in > replicate-distribute with each 4 bricks for 6 of them and 2 bricks for the > remaining 2. Full healing will just take ages... for a just single brick to > resync ! > >> gluster v status home > volume status home > Status of volume: home > Gluster process TCP Port RDMA Port Online Pid > -- > Brick server1:/data/glusterfs/home/brick1 49157 0 Y 5003 > Brick server1:/data/glusterfs/home/brick2 49153 0 Y 5023 > Brick server1:/data/glusterfs/home/brick3 49154 0 Y 5004 > Brick server1:/data/glusterfs/home/brick4 49155 0 Y 5011 > Brick server3:/data/glusterfs/home/brick1 49152 0 Y 5422 > Brick server4:/data/glusterfs/home/brick1 49152 0 Y 5019 > Brick server3:/data/glusterfs/home/brick2 49153 0 Y 5429 > Brick server4:/data/glusterfs/home/brick2 49153 0 Y 5033 > Brick server3:/data/glusterfs/home/brick3 49154 0 Y 5437 > Brick server4:/data/glusterfs/home/brick3 49154 0 Y 5026 > Brick server3:/data/glusterfs/home/brick4 49155 0 Y 5444 > Brick server4:/data/glusterfs/home/brick4 N/A N/AN N/A > Brick server5:/data/glusterfs/home/brick1 49152 0 Y 5275 > Brick server6:/data/glusterfs/home/brick1 49152 0 Y 5786 > Brick server5:/data/glusterfs/home/brick2 49153 0 Y 5276 > Brick server6:/data/glusterfs/home/brick2 49153 0 Y 5792 > Brick server5:/data/glusterfs/home/brick3 49154 0 Y 5282 > Brick server6:/data/glusterfs/home/brick3 49154 0 Y 5794 > Brick server5:/data/glusterfs/home/brick4 49155 0 Y 5293 > Brick server6:/data/glusterfs/home/brick4 49155 0 Y 5806 > Brick server7:/data/glusterfs/home/brick1 49156 0 Y 22339 > Brick server8:/data/glusterfs/home/brick1 49153 0 Y 17992 > Brick server7:/data/glusterfs/home/brick2 49157 0 Y 22347 > Brick server8:/data/glusterfs/home/brick2 49154 0 Y 18546 > NFS Server on localhost 2049 0 Y 683 > Self-heal Daemon on localhost N/A N/AY 693 > NFS Server on server8 2049 0 Y 18553 > Self-heal Daemon on server8N/A N/AY 18566 > NFS Server on server5 2049 0 Y 23115 > Self-heal Daemon on server5N/A N/AY 23121 > NFS Server on server7 2049 0 Y 4201 > Self-heal Daemon on server7N/A N/AY 4210 > NFS Server on server3 2049 0 Y 5460 > Self-heal Daemon on server3N/A N/AY 5469 > NFS Server on server6 2049 0 Y 22709 > Self-heal Daemon on server6N/A N/AY 22718 > NFS Server on server4 2049 0 Y 6044 > Self-heal Daemon on server4N/A N/AY 6243 > > server 2 is currently powered off as we are waiting a replacement RAID > controller, as well as for > server4:/data/glusterfs/home/brick4 > > And as I said, there is a rebalance in progress >> gluster rebalance home status > Node Rebalanced-files size > scanned failures skipped status run time in h:m:s >- --- --- > --- --- --- > -- >localhost4208323.3GB > 1568065 1359303734 in progress 16:49:31 > server53569823.8GB > 1027934 0240748 in progress 16:49:23 > server43509623.4GB > 899491 0229064 in progress 16:49:18 > server32703118.0GB > 701759 8182592
Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?
Hi, I simplified the config in my first email, but I actually have 2x4 servers in replicate-distribute with each 4 bricks for 6 of them and 2 bricks for the remaining 2. Full healing will just take ages... for a just single brick to resync ! > gluster v status home volume status home Status of volume: home Gluster process TCP Port RDMA Port Online Pid -- Brick server1:/data/glusterfs/home/brick1 49157 0 Y 5003 Brick server1:/data/glusterfs/home/brick2 49153 0 Y 5023 Brick server1:/data/glusterfs/home/brick3 49154 0 Y 5004 Brick server1:/data/glusterfs/home/brick4 49155 0 Y 5011 Brick server3:/data/glusterfs/home/brick1 49152 0 Y 5422 Brick server4:/data/glusterfs/home/brick1 49152 0 Y 5019 Brick server3:/data/glusterfs/home/brick2 49153 0 Y 5429 Brick server4:/data/glusterfs/home/brick2 49153 0 Y 5033 Brick server3:/data/glusterfs/home/brick3 49154 0 Y 5437 Brick server4:/data/glusterfs/home/brick3 49154 0 Y 5026 Brick server3:/data/glusterfs/home/brick4 49155 0 Y 5444 Brick server4:/data/glusterfs/home/brick4 N/A N/AN N/A Brick server5:/data/glusterfs/home/brick1 49152 0 Y 5275 Brick server6:/data/glusterfs/home/brick1 49152 0 Y 5786 Brick server5:/data/glusterfs/home/brick2 49153 0 Y 5276 Brick server6:/data/glusterfs/home/brick2 49153 0 Y 5792 Brick server5:/data/glusterfs/home/brick3 49154 0 Y 5282 Brick server6:/data/glusterfs/home/brick3 49154 0 Y 5794 Brick server5:/data/glusterfs/home/brick4 49155 0 Y 5293 Brick server6:/data/glusterfs/home/brick4 49155 0 Y 5806 Brick server7:/data/glusterfs/home/brick1 49156 0 Y 22339 Brick server8:/data/glusterfs/home/brick1 49153 0 Y 17992 Brick server7:/data/glusterfs/home/brick2 49157 0 Y 22347 Brick server8:/data/glusterfs/home/brick2 49154 0 Y 18546 NFS Server on localhost 2049 0 Y 683 Self-heal Daemon on localhost N/A N/AY 693 NFS Server on server8 2049 0 Y 18553 Self-heal Daemon on server8N/A N/AY 18566 NFS Server on server5 2049 0 Y 23115 Self-heal Daemon on server5N/A N/AY 23121 NFS Server on server7 2049 0 Y 4201 Self-heal Daemon on server7N/A N/AY 4210 NFS Server on server3 2049 0 Y 5460 Self-heal Daemon on server3N/A N/AY 5469 NFS Server on server6 2049 0 Y 22709 Self-heal Daemon on server6N/A N/AY 22718 NFS Server on server4 2049 0 Y 6044 Self-heal Daemon on server4N/A N/AY 6243 server 2 is currently powered off as we are waiting a replacement RAID controller, as well as for server4:/data/glusterfs/home/brick4 And as I said, there is a rebalance in progress > gluster rebalance home status Node Rebalanced-files size scanned failures skipped status run time in h:m:s - --- --- --- --- --- -- localhost4208323.3GB 1568065 1359303734 in progress 16:49:31 server53569823.8GB 1027934 0240748 in progress 16:49:23 server43509623.4GB 899491 0229064 in progress 16:49:18 server32703118.0GB 701759 8182592 in progress 16:49:27 server800Bytes 327602 0 805 in progress 16:49:18 server63567223.9GB 1028469 0240810 in progress 16:49:17 server71 45Bytes 53 0 0complete
Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?
Hi, Thanks. However "gluster v heal volname full" returned the following error message Commit failed on server4. Please check log file for details. I have checked the log files in /var/log/glusterfs on server4 (by grepping heal), but did not get any match. What should I be looking for and in which log file, please ? Note that there is currently a rebalance process running on the volume. Many thanks, A. On Thursday, 1 February 2018 17:32:19 CET Serkan Çoban wrote: > You do not need to reset brick if brick path does not change. Replace > the brick format and mount, then gluster v start volname force. > To start self heal just run gluster v heal volname full. > > On Thu, Feb 1, 2018 at 6:39 PM, Alessandro Ipe wrote: > > Hi, > > > > > > My volume home is configured in replicate mode (version 3.12.4) with the > > bricks server1:/data/gluster/brick1 > > server2:/data/gluster/brick1 > > > > server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon for > > that brick on server2, umounted it, reformated it, remounted it and did a> > >> gluster volume reset-brick home server2:/data/gluster/brick1 > >> server2:/data/gluster/brick1 commit force> > > I was expecting that the self-heal daemon would start copying data from > > server1:/data/gluster/brick1 (about 7.4 TB) to the empty > > server2:/data/gluster/brick1, which it only did for directories, but not > > for files. > > > > For the moment, I launched on the fuse mount point > > > >> find . | xargs stat > > > > but crawling the whole volume (100 TB) to trigger self-healing of a single > > brick of 7.4 TB is unefficient. > > > > Is there any trick to only self-heal a single brick, either by setting > > some attributes to its top directory, for example ? > > > > > > Many thanks, > > > > > > Alessandro > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users -- Dr. Ir. Alessandro Ipe Department of Observations Tel. +32 2 373 06 31 Remote Sensing from Space Royal Meteorological Institute Avenue Circulaire 3Email: B-1180 BrusselsBelgium alessandro@meteo.be Web: http://gerb.oma.be ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to trigger a resync of a newly replaced empty brick in replicate config ?
You do not need to reset brick if brick path does not change. Replace the brick format and mount, then gluster v start volname force. To start self heal just run gluster v heal volname full. On Thu, Feb 1, 2018 at 6:39 PM, Alessandro Ipe wrote: > Hi, > > > My volume home is configured in replicate mode (version 3.12.4) with the > bricks > server1:/data/gluster/brick1 > server2:/data/gluster/brick1 > > server2:/data/gluster/brick1 was corrupted, so I killed gluster daemon for > that brick on server2, umounted it, reformated it, remounted it and did a >> gluster volume reset-brick home server2:/data/gluster/brick1 >> server2:/data/gluster/brick1 commit force > > I was expecting that the self-heal daemon would start copying data from > server1:/data/gluster/brick1 > (about 7.4 TB) to the empty server2:/data/gluster/brick1, which it only did > for directories, but not for files. > > For the moment, I launched on the fuse mount point >> find . | xargs stat > but crawling the whole volume (100 TB) to trigger self-healing of a single > brick of 7.4 TB is unefficient. > > Is there any trick to only self-heal a single brick, either by setting some > attributes to its top directory, for example ? > > > Many thanks, > > > Alessandro > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users