Here is one test performed on a 300GB data set and around 100%(1/2 the time)
improvement was seen.
[root@gprfs031 ~]# gluster v i
Volume Name: rbperf
Type: Distribute
Volume ID: 35562662-337e-4923-b862-d0bbb0748003
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: gprfs029-10ge:/bricks/gprfs029/brick1
Brick2: gprfs030-10ge:/bricks/gprfs030/brick1
Brick3: gprfs031-10ge:/bricks/gprfs031/brick1
Brick4: gprfs032-10ge:/bricks/gprfs032/brick1
Added server 32 and started rebalance force.
Rebalance stat for new changes:
[root@gprfs031 ~]# gluster v rebalance rbperf status
Node Rebalanced-files size
scanned failures skipped status run time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 74639 36.1GB
297319 0 0 completed 1743.00
172.17.40.30 67512 33.5GB
269187 0 0 completed 1395.00
gprfs029-10ge 79095 38.8GB
284105 0 0 completed 1559.00
gprfs032-10ge 0 0Bytes
0 0 0 completed 402.00
volume rebalance: rbperf: success:
Rebalance stat for old model:
[root@gprfs031 ~]# gluster v rebalance rbperf status
Node Rebalanced-files size
scanned failures skipped status run time in secs
--------- ----------- -----------
----------- ----------- ----------- ------------ --------------
localhost 86493 42.0GB
634302 0 0 completed 3329.00
gprfs029-10ge 94115 46.2GB
687852 0 0 completed 3328.00
gprfs030-10ge 74314 35.9GB
651943 0 0 completed 3072.00
gprfs032-10ge 0 0Bytes
594166 0 0 completed 1943.00
volume rebalance: rbperf: success:
Best regards,
Susant
----- Original Message -----
From: "Vijay Bellur" <[email protected]>
To: "Susant Palai" <[email protected]>, "Gluster Devel"
<[email protected]>
Sent: Monday, 6 April, 2015 7:39:27 PM
Subject: Re: [Gluster-devel] Rebalance improvement design
On 03/31/2015 01:49 PM, Susant Palai wrote:
> Hi,
> Posted patch for rebalance improvement here:
> http://review.gluster.org/#/c/9657/ .
> You can find the feature page here:
> http://www.gluster.org/community/documentation/index.php/Features/improve_rebalance_performance
>
> The current patch address two part of the design proposed.
> 1. Rebalance multiple files in parallel
> 2. Crawl only bricks that belong to the current node
>
> Brief design explanation for the above two points.
>
> 1. Rebalance multiple files in parallel:
> -------------------------------------
>
> The existing rebalance engine is single threaded. Hence, introduced
> multiple threads which will be running parallel to the crawler.
> The current rebalance migration is converted to a
> "Producer-Consumer" frame work.
> Where Producer is : Crawler
> Consumer is : Migrating Threads
>
> Crawler: Crawler is the main thread. The job of the crawler is now
> limited to fix-layout of each directory and add the files
> which are eligible for the migration to a global queue.
> Hence, the crawler will not be "blocked" by migration process.
>
> Producer: Producer will monitor the global queue. If any file is
> added to this queue, it will dqueue that entry and migrate the file.
> Currently 15 migration threads are spawned at the beginning
> of the rebalance process. Hence, multiple file migration
> happens in parallel.
>
>
> 2. Crawl only bricks that belong to the current node:
> --------------------------------------------------
>
> As rebalance process is spawned per node, it migrates only the
> files that belongs to it's own node for the sake of load
> balancing. But it also reads entries from the whole cluster,
> which is not necessary as readdir hits other nodes.
>
> New Design:
> As part of the new design the rebalancer decides the subvols that
> are local to the rebalancer node by checking the node-uuid of
> root directory prior to the crawler starts. Hence, readdir won't
> hit the whole cluster as it has already the context of
> local subvols and also node-uuid request for each file can be
> avoided. This makes the rebalance process "more scalable".
>
>
The approaches outlined do look good.
Do you have rebalance comparison numbers before and after this patchset?
Thanks,
Vijay
_______________________________________________
Gluster-devel mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-devel
