Hi Susant, Thanks for your reply.I think, we started the re-balance with force option. I requested a question on this mail thread regarding to run the rebalancer daemon forcefully on a dedicated peered node instead of selecting a cluster node automatically.
If i start rebalancer, the rebalancer daemon starts the fixlayout on all nodes and migrate files on anyone node(always node 1 on our cluster), The first node cpu usage is always high during rebalance compare with rest of the cluster nodes. To reduce the cpu usage of rebalancer datanode( node1), i peered a new node( without disk) for re-balance and started the rebalancer. It started again the rebalancer on same node1. Is there any way to run the rebalancer daemon( file migration) forcefully on a dedicated peered node? Regards Backer On Thu, Aug 27, 2015 at 3:00 PM, Susant Palai <[email protected]> wrote: > comments inline. > ++Ccing Pranith and Ashish to detail on disperse behaviour. > > ----- Original Message ----- > From: "Mohamed Pakkeer" <[email protected]> > To: "Susant Palai" <[email protected]>, "Vijay Bellur" <[email protected] > > > Cc: "Mathieu Chateau" <[email protected]>, "gluster-users" < > [email protected]>, "Gluster Devel" <[email protected]> > Sent: Wednesday, 26 August, 2015 2:08:02 PM > Subject: Re: [Gluster-users] cluster.min-free-disk is not working in > distributed disperse volume > > > Hi Susant, > > > > I am monitoring the re-balance daemon data movement. I have few queries > about this data movement.I can see the data movement on the following cases. > > > 1. Source sub-volume is 90% occupied and destination sub-volume is 98% > occupied. > log info > [2015-08-26 06:53:26.219674] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-qubevaultdr-dht: completed > migration of > /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf > from subvolume qubevaultdr-disperse-7 to qubevaultdr-disperse-10 > > Can you mention whether you gave force option while starting rebalance? If > force flag is set, rebalance will move the data with out considering > whether it is moving the data from a high avail space to a low avail space. > > > 2. Source sub-volume is 98% occupied and destination sub-volume is 97% > occupied > log info > [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf: > attempting to move from glustertest-disperse-10 to glustertest-disperse-8 > > > 3. source sub-volume is 94% occupied and destination sub-volume is 15% > occupied > log info > [2015-08-26 06:53:26.348804] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP > from sub-volume glustertest-disperse-26 to glustertest-disperse-56 > > > I can see the case 2 and 3 are valid cases because source has more data > than destination. But Case 1 is invalid, because source sub-volume has less > data than destination sub-volume. Also i have one more query about case 2. > Why does the case 2 choose the destination as 94% occupied sub-volume > instead of so many under utilized sub-volumes( 13% to 15 % sub-volumes- > glustertest-disperse-36 to glustertest-disperse-71 ) ? > > > > > Rebalance Log > > > [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf: > attempting to move from glustertest-disperse-10 to glustertest-disperse-8 > [2015-08-26 06:52:22.633777] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X > X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER_SMK_MPEG_260514-reel-1-mpeg2.mxf > from subvolume glustertest-disperse-9 to glustertest-disperse-34 > [2015-08-26 06:52:22.638955] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-3-mpeg2.mxf: > attempting to move from glustertest-disperse-14 to glustertest-disperse-39 > [2015-08-26 06:52:50.171304] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X > X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf > from sub-volume glustertest-disperse-10 to glustertest-disperse-8 > [2015-08-26 06:52:50.179238] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-4-mpeg2.mxf: > attempting to move from glustertest-disperse-16 to glustertest-disperse-5 > [2015-08-26 06:53:08.579988] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of /Packages/Features/MPEG/B/BrokenHorses_FTR_S > _EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-4-mpeg2.mxf > from subvolume glustertest-disperse-2 to glustertest-disperse-23 > [2015-08-26 06:53:08.583021] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log: > attempting to move from glustertest-disperse-18 to glustertest-disperse-0 > [2015-08-26 06:53:08.698014] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X > X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log > from sub-volume glustertest-disperse-18 to glustertest-disperse-0 > [2015-08-26 06:53:08.703567] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_ > S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-5-mpeg2.mxf: > attempting to move from glustertest-disperse-21 to glustertest-disperse-40 > [2015-08-26 06:53:26.219674] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of > /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf > from subvolume glustertest-disperse-7 to glustertest-disperse-10 > [2015-08-26 06:53:26.226307] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP: > attempting to move from glustertest-disperse-26 to glustertest-disperse-56 > [2015-08-26 06:53:26.348804] I [MSGID: 109022] > [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed > migration of > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP > from subvolume glustertest-disperse-26 to glustertest-disperse-56 > [2015-08-26 06:53:26.354215] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R8_MPEG_AUDIO_250514.mxf: > attempting to move from glustertest-disperse-29 to glustertest-disperse-16 > > > > > Subvolume0 to subvolume35 > > > root@glustertestdn001:# df -h > Filesystem Size Used Avail Use% Mounted on Subvolume Details > ========== ==== ==== ===== ==== =========== ================== > > > /dev/sdb1 3.7T 3.6T 51G 99% /media/disk1 - glustertest-disperse-0 > /dev/sdc1 3.7T 3.3T 384G 90% /media/disk2 - glustertest-disperse-1 > /dev/sdd1 3.7T 3.6T 76G 98% /media/disk3 - glustertest-disperse-2 > /dev/sde1 3.7T 3.5T 214G 95% /media/disk4 - glustertest-disperse-3 > /dev/sdf1 3.7T 3.3T 365G 91% /media/disk5 - glustertest-disperse-4 > /dev/sdg1 3.7T 3.5T 173G 96% /media/disk6 - glustertest-disperse-5 > /dev/sdh1 3.7T 3.4T 284G 93% /media/disk7 - glustertest-disperse-6 > /dev/sdi1 3.7T 3.3T 373G 90% /media/disk8 - glustertest-disperse-7 > /dev/sdj1 3.7T 3.6T 122G 97% /media/disk9 - glustertest-disperse-8 > /dev/sdk1 3.7T 3.4T 247G 94% /media/disk10 - glustertest-disperse-9 > /dev/sdl1 3.7T 3.6T 108G 98% /media/disk11 - glustertest-disperse-10 > /dev/sdm1 3.7T 3.4T 252G 94% /media/disk12 - glustertest-disperse-11 > /dev/sdn1 3.7T 3.5T 241G 94% /media/disk13 - glustertest-disperse-12 > /dev/sdo1 3.7T 3.6T 136G 97% /media/disk14 - glustertest-disperse-13 > /dev/sdp1 3.7T 3.4T 247G 94% /media/disk15 - glustertest-disperse-14 > /dev/sdq1 3.7T 3.5T 170G 96% /media/disk16 - glustertest-disperse-15 > /dev/sdr1 3.7T 3.4T 257G 94% /media/disk17 - glustertest-disperse-16 > /dev/sds1 3.7T 3.4T 258G 94% /media/disk18 - glustertest-disperse-17 > /dev/sdt1 3.7T 3.5T 171G 96% /media/disk19 - glustertest-disperse-18 > /dev/sdu1 3.7T 3.5T 184G 96% /media/disk20 - glustertest-disperse-19 > /dev/sdv1 3.7T 3.4T 281G 93% /media/disk21 - glustertest-disperse-20 > /dev/sdw1 3.7T 3.3T 347G 91% /media/disk22 - glustertest-disperse-21 > /dev/sdx1 3.7T 3.5T 191G 95% /media/disk23 - glustertest-disperse-22 > /dev/sdy1 3.7T 3.5T 217G 95% /media/disk24 - glustertest-disperse-23 > /dev/sdz1 3.7T 3.4T 263G 93% /media/disk25 - glustertest-disperse-24 > /dev/sdaa1 3.7T 3.6T 66G 99% /media/disk26 - glustertest-disperse-25 > /dev/sdab1 3.7T 3.5T 225G 94% /media/disk27 - glustertest-disperse-26 > /dev/sdac1 3.7T 3.4T 310G 92% /media/disk28 - glustertest-disperse-27 > /dev/sdad1 3.7T 3.5T 159G 96% /media/disk29 - glustertest-disperse-28 > /dev/sdae1 3.7T 3.3T 373G 91% /media/disk30 - glustertest-disperse-29 > /dev/sdaf1 3.7T 3.4T 305G 92% /media/disk31 - glustertest-disperse-30 > /dev/sdag1 3.7T 3.5T 184G 96% /media/disk32 - glustertest-disperse-31 > /dev/sdah1 3.7T 3.5T 186G 96% /media/disk33 - glustertest-disperse-32 > /dev/sdai1 3.7T 3.3T 373G 91% /media/disk34 - glustertest-disperse-33 > /dev/sdaj1 3.7T 3.3T 361G 91% /media/disk35 - glustertest-disperse-34 > /dev/sdak1 3.7T 3.3T 375G 90% /media/disk36 - glustertest-disperse-35 > > > > > > > Subvolume36 to subvolume71 > > > > > > root@glustertestdn011:/var/log/glusterfs# df -h > > > Filesystem Size Used Avail Use% Mounted on SubVolume Detail > ========== ==== ==== ===== ==== ========== ================ > /dev/sdc1 3.7T 475G 3.2T 13% /media/disk1 - glustertest-disperse-36 > /dev/sdd1 3.7T 467G 3.2T 13% /media/disk2 - glustertest-disperse-37 > /dev/sde1 3.7T 492G 3.2T 14% /media/disk3 - glustertest-disperse-38 > /dev/sdf1 3.7T 470G 3.2T 13% /media/disk4 - glustertest-disperse-39 > /dev/sdg1 3.7T 498G 3.2T 14% /media/disk5 - glustertest-disperse-40 > /dev/sdh1 3.7T 461G 3.2T 13% /media/disk6 - glustertest-disperse-41 > /dev/sdi1 3.7T 451G 3.2T 13% /media/disk7 - glustertest-disperse-42 > /dev/sdj1 3.7T 464G 3.2T 13% /media/disk8 - glustertest-disperse-43 > /dev/sdk1 3.7T 426G 3.3T 12% /media/disk9 - glustertest-disperse-44 > /dev/sdl1 3.7T 425G 3.3T 12% /media/disk10 - glustertest-disperse-45 > /dev/sdm1 3.7T 460G 3.2T 13% /media/disk11 - glustertest-disperse-46 > /dev/sdn1 3.7T 478G 3.2T 13% /media/disk12 - glustertest-disperse-47 > /dev/sdo1 3.7T 475G 3.2T 13% /media/disk13 - glustertest-disperse-48 > /dev/sdp1 3.7T 520G 3.2T 14% /media/disk14 - glustertest-disperse-49 > /dev/sdq1 3.7T 456G 3.2T 13% /media/disk15 - glustertest-disperse-50 > /dev/sdr1 3.7T 442G 3.3T 12% /media/disk16 - glustertest-disperse-51 > /dev/sds1 3.7T 494G 3.2T 14% /media/disk17 - glustertest-disperse-52 > /dev/sdt1 3.7T 495G 3.2T 14% /media/disk18 - glustertest-disperse-53 > /dev/sdu1 3.7T 426G 3.3T 12% /media/disk19 - glustertest-disperse-54 > /dev/sdv1 3.7T 471G 3.2T 13% /media/disk20 - glustertest-disperse-55 > /dev/sdw1 3.7T 531G 3.2T 15% /media/disk21 - glustertest-disperse-56 > /dev/sdx1 3.7T 576G 3.1T 16% /media/disk22 - glustertest-disperse-57 > /dev/sdy1 3.7T 411G 3.3T 12% /media/disk23 - glustertest-disperse-58 > /dev/sdz1 3.7T 455G 3.2T 13% /media/disk24 - glustertest-disperse-59 > /dev/sdaa1 3.7T 427G 3.3T 12% /media/disk25 - glustertest-disperse-60 > /dev/sdab1 3.7T 442G 3.3T 12% /media/disk26 - glustertest-disperse-61 > /dev/sdac1 3.7T 449G 3.2T 13% /media/disk27 - glustertest-disperse-62 > /dev/sdad1 3.7T 446G 3.3T 12% /media/disk28 - glustertest-disperse-63 > /dev/sdae1 3.7T 412G 3.3T 12% /media/disk29 - glustertest-disperse-64 > /dev/sdaf1 3.7T 480G 3.2T 13% /media/disk30 - glustertest-disperse-65 > /dev/sdag1 3.7T 485G 3.2T 14% /media/disk31 - glustertest-disperse-66 > /dev/sdah1 3.7T 453G 3.2T 13% /media/disk32 - glustertest-disperse-67 > /dev/sdai1 3.7T 410G 3.3T 12% /media/disk33 - glustertest-disperse-68 > /dev/sdaj1 3.7T 457G 3.2T 13% /media/disk34 - glustertest-disperse-69 > /dev/sdak1 3.7T 444G 3.3T 12% /media/disk35 - glustertest-disperse-70 > /dev/sdal1 3.7T 484G 3.2T 13% /media/disk36 - glustertest-disperse-71 > > > > > Is this the normal behavior of DHT in distributed disperse volume ? > > Requesting Pranith and Ashish to comment on this. > > Thanks > Backer > > > > > > > > On Tue, Aug 25, 2015 at 12:36 PM, Susant Palai < [email protected] > > wrote: > > > Mohamed, > Will investigate in to weighted rebalance behavior. > > Susant > > ----- Original Message ----- > From: "Mohamed Pakkeer" < [email protected] > > To: "Susant Palai" < [email protected] > > Cc: "Mathieu Chateau" < [email protected] >, "gluster-users" < > [email protected] >, "Gluster Devel" < [email protected] > > Sent: Tuesday, 25 August, 2015 9:40:01 AM > Subject: Re: [Gluster-users] cluster.min-free-disk is not working in > distributed disperse volume > > > > > Hi Sasant, > We have created the disperse volume across nodes. We stopped all the > upload operations and started the rebalance last night.After overnight > re-balance, some harddisk is occupied 100% and some disks have 13% disk > space. > > > disk1 belongs to disperse-set-0 ..... disk36 belongs to disperse-set-35 > > > df -h result of one data node > > > > /dev/sdb1 3.7T 3.7T 545M 100% /media/disk1 > /dev/sdc1 3.7T 3.2T 496G 87% /media/disk2 > /dev/sdd1 3.7T 3.7T 30G 100% /media/disk3 > /dev/sde1 3.7T 3.5T 173G 96% /media/disk4 > /dev/sdf1 3.7T 3.2T 458G 88% /media/disk5 > /dev/sdg1 3.7T 3.5T 143G 97% /media/disk6 > /dev/sdh1 3.7T 3.5T 220G 95% /media/disk7 > /dev/sdi1 3.7T 3.3T 415G 89% /media/disk8 > /dev/sdj1 3.7T 3.6T 72G 99% /media/disk9 > /dev/sdk1 3.7T 3.5T 186G 96% /media/disk10 > /dev/sdl1 3.7T 3.6T 65G 99% /media/disk11 > /dev/sdm1 3.7T 3.5T 195G 95% /media/disk12 > /dev/sdn1 3.7T 3.5T 199G 95% /media/disk13 > /dev/sdo1 3.7T 3.6T 78G 98% /media/disk14 > /dev/sdp1 3.7T 3.5T 200G 95% /media/disk15 > /dev/sdq1 3.7T 3.6T 119G 97% /media/disk16 > /dev/sdr1 3.7T 3.5T 206G 95% /media/disk17 > /dev/sds1 3.7T 3.5T 193G 95% /media/disk18 > /dev/sdt1 3.7T 3.6T 131G 97% /media/disk19 > /dev/sdu1 3.7T 3.5T 141G 97% /media/disk20 > /dev/sdv1 3.7T 3.5T 243G 94% /media/disk21 > /dev/sdw1 3.7T 3.4T 299G 92% /media/disk22 > /dev/sdx1 3.7T 3.5T 163G 96% /media/disk23 > /dev/sdy1 3.7T 3.5T 168G 96% /media/disk24 > /dev/sdz1 3.7T 3.5T 219G 95% /media/disk25 > /dev/sdaa1 3.7T 3.7T 37G 100% /media/disk26 > /dev/sdab1 3.7T 3.5T 172G 96% /media/disk27 > /dev/sdac1 3.7T 3.4T 276G 93% /media/disk28 > /dev/sdad1 3.7T 3.6T 108G 98% /media/disk29 > /dev/sdae1 3.7T 3.3T 399G 90% /media/disk30 > /dev/sdaf1 3.7T 3.5T 240G 94% /media/disk31 > /dev/sdag1 3.7T 3.6T 122G 97% /media/disk32 > /dev/sdah1 3.7T 3.5T 147G 97% /media/disk33 > /dev/sdai1 3.7T 3.4T 342G 91% /media/disk34 > /dev/sdaj1 3.7T 3.4T 288G 93% /media/disk35 > /dev/sdak1 3.7T 3.4T 342G 91% /media/disk36 > > > disk1 belongs to disperse-set-0. Rebalancer logs shows, still rebalancer > is trying to fill the disperse-set-0 after filling to 100% > > > > > [2015-08-24 19:52:53.036622] E [MSGID: 109023] > [dht-rebalance.c:672:__dht_check_free_space] 0-glustertest-dht: data > movement attempted from node (glustertest-disperse-7) to node > (glustertest-disperse-0) which does not have required free space for > (/Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMI-NEBO-C_R3_AUDIO_190511.mxf) > > > [2015-08-24 19:52:53.042026] I [dht-rebalance.c:1002:dht_migrate_file] > 0-glustertest-dht: > /Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMINEBO-CHALLANGE_BEN_R1-2-3-4-5-6_MPEG_200511-reel-5-mpeg2.mxf: > attempting to move from glustertest-disperse-13 to glustertest-disperse-0 > > > I think, cluster.weighted-rebalance and cluster.min-free-dis k have bugs > for re-balancing the data, based on weight and disk free space. > > > > > Thanks > Backer > > > > > On Mon, Aug 24, 2015 at 4:28 PM, Mohamed Pakkeer < [email protected] > > wrote: > > > > Hi Susant, > > > Thanks for your quick reply. We are not updating any files. Actually we > are archiving video files on this cluster. I think there is a bug in > cluster.min-free-disk. > > > Also i would like to know about rebalance the cluster. Currently we have > 20 nodes and 10 nodes hard disks are almost full . So we need to rebalance > the data. If i run the rebalancer, it starts on first node(node1) and > starts the migration process. The first node cpu usage is always high > during rebalance compare with rest of the cluster nodes.To reduce the cpu > usage of rebalancer datanode( node1), i peer a new node( without disk) for > rebalance and start the rebalancer. It starts again the rebalancer on same > node1. How can we run a rebalancer on a dedicated node? > > > Also we are facing memory leaks in fixlayout and heal full operations. > > > Regards > Backer > > > > > On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai < [email protected] > wrote: > > > Hi, > Cluster.min-free-disk controls new file creation on the bricks. If you > happen to write to the existing files on the brick and that is leading to > brick getting full, then most probably you should run a rebalance. > > Regards, > Susant > > > > ----- Original Message ----- > From: "Mathieu Chateau" < [email protected] > > To: "Mohamed Pakkeer" < [email protected] > > Cc: "gluster-users" < [email protected] >, "Gluster Devel" < > [email protected] > > Sent: Monday, 24 August, 2015 2:47:00 PM > Subject: Re: [Gluster-users] cluster.min-free-disk is not working in > distributed disperse volume > > > > > 720 brick! Respect ! > Le 24 août 2015 09:48, "Mohamed Pakkeer" < [email protected] > a écrit : > > > > Hi, > > > I have a cluster of 720 bricks, all bricks are 4TB in size. I have change > the cluster.min-free-disk default value 10% to 3%. So all the disks should > have 3% minimum disk space free. But some cluster disks are getting full > now. Is there any additional configuration for keeping some percentage of > disk space kept free? > > > > > > Volume Name: glustertest > Type: Distributed-Disperse > Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609 > Status: Started > Number of Bricks: 72 x (8 + 2) = 720 > Transport-type: tcp > > > > > > Options Reconfigured: > features.default-soft-limit: 95% > cluster.min-free-disk: 3% > performance.readdir-ahead: on > > > df -h of one node > > > > /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1 > /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2 > /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3 > > > Any help will be greatly appreciated. > > > > > Regards > Backer > > > > > > > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > -- > > Thanks & Regards > K. Mohamed Pakkeer > Mobile- 0091-8754410114 > >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
