Thanks Vijay! I forgot to upgrade the kernel(thinp 6.6 perf bug gah) before I created this data set, so its a bit smaller:
total threads = 16 total files = 7,060,700 (64 kb files, 100 files per dir) total data = 430.951 GB 88.26% of requested files processed, minimum is 70.00 10101.355737 sec elapsed time 698.985382 files/sec 698.985382 IOPS 43.686586 MB/sec I updated everything and ran the rebalanace on glusterfs-3.8dev-0.107.git275f724.el6.x86_64.: [root@gqas001 ~]# gluster v rebalance testvol status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 1327346 81.0GB 3999140 0 0 completed 55088.00 gqas013.sbu.lab.eng.bos.redhat.com 0 0Bytes 1 0 0 completed 26070.00 gqas011.sbu.lab.eng.bos.redhat.com 0 0Bytes 0 0 0 failed 0.00 gqas014.sbu.lab.eng.bos.redhat.com 0 0Bytes 0 0 0 failed 0.00 gqas016.sbu.lab.eng.bos.redhat.com 1325857 80.9GB 4000865 0 0 completed 55088.00 gqas015.sbu.lab.eng.bos.redhat.com 0 0Bytes 0 0 0 failed 0.00 volume rebalance: testvol: success: A couple observations: I am seeing lots of threads / processes running: [root@gqas001 ~]# ps -eLf | grep glu | wc -l 96 <- 96 gluster threads [root@gqas001 ~]# ps -eLf | grep rebal | wc -l 36 <- 36 rebal threads. Is this tunible? Is there a use case where we would need to limit this? Just curious, how did we arrive at 36 rebal threads? # cat /var/log/glusterfs/testvol-rebalance.log | wc -l 4,577,583 [root@gqas001 ~]# ll /var/log/glusterfs/testvol-rebalance.log -h -rw------- 1 root root 1.6G May 3 12:29 /var/log/glusterfs/testvol-rebalance.log :) How big is this going to get when I do the 10-20 TB? I'll keep tabs on this, my default test setup only has: [root@gqas001 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_gqas001-lv_root 50G 4.8G 42G 11% / tmpfs 24G 0 24G 0% /dev/shm /dev/sda1 477M 65M 387M 15% /boot /dev/mapper/vg_gqas001-lv_home 385G 71M 366G 1% /home /dev/mapper/gluster_vg-lv_bricks 9.5T 219G 9.3T 3% /bricks Next run I want to fill up a 10TB cluster and double the # of bricks to simulate running out of space doubling capacity. Any other fixes or changes that need to go in before I try a larger data set? Before that I may run my performance regression suite against a system while a rebal is in progress and check how it affects performance. I'll turn both these cases into perf regression tests that I run with iozone smallfile and such, any other use cases I should add? Should I add hard / soft links / whatever else tot he data set? -b On Sun, May 3, 2015 at 11:48 AM, Vijay Bellur <vbel...@redhat.com> wrote: > On 05/01/2015 10:23 AM, Benjamin Turner wrote: > >> Ok I have all my data created and I just started the rebalance. One >> thing to not in the client log I see the following spamming: >> >> [root@gqac006 ~]# cat /var/log/glusterfs/gluster-mount-.log | wc -l >> 394042 >> >> [2015-05-01 00:47:55.591150] I [MSGID: 109036] >> [dht-common.c:6478:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht: >> Setting layout of >> /file_dstdir/ >> gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006 >> < >> http://gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006 >> > >> with [Subvol_name: testvol-replicate-0, Err: -1 , Start: 0 , Stop: >> 2141429669 ], [Subvol_name: testvol-replicate-1, Err: -1 , Start: >> 2141429670 , Stop: 4294967295 ], >> [2015-05-01 00:47:55.596147] I >> [dht-selfheal.c:1587:dht_selfheal_layout_new_directory] 0-testvol-dht: >> chunk size = 0xffffffff / 19920276 = 0xd7 >> [2015-05-01 00:47:55.596177] I >> [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht: >> assigning range size 0x7fa39fa6 to testvol-replicate-1 >> > > > I also noticed the same set of excessive logs in my tests. Have sent > across a patch [1] to address this problem. > > -Vijay > > [1] http://review.gluster.org/10281 > > > >
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel