[
https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13708498#comment-13708498
]
Jean-Marc Spaggiari commented on HBASE-8803:
--------------------------------------------
(Sorry, long comment because of all those logs)
When regions are unload from the server to be assigned somewhere else, threads
are limited to the number of servers. So if you have only 9 nodes on your
cluster, then your server can send only to 8 other nodes, and that'S why you
have a 8 limitation.
When regions are re-loaded from the other nodes to the server, then the only
limitation is the thread parameter. Below are some tests for this scenario.
{code}
13/07/15 09:38:39 INFO region_mover: Moving region
83a139f0629f2afbccc16f28c1fc5b07 (1 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
20a2b8e4d5346d05fc5659a04d5ccc2d (2 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
7b4dabe9857fdec833791bf70a6446a0 (3 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
7e1feaaff3c1be8e413072b4671245bf (4 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
e6d0569bd931317429cb5a130ea6a7f4 (5 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
d2190b8d656d6a813096b8d7b85f124c (6 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
1c024b8cabfea6dd735c6444cc7237a0 (7 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
0dfe044b3625984954505cfdf7adb32d (8 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
71c9d0ba64b16fb18fa7907227cb866a (9 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:39 INFO region_mover: Moving region
1ccf69eb2e53c0decaf2dfa7aca86257 (10 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
8f5afbea06948fd57014bab1e9e07ea3 (11 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
0f08b0a2377ec8633259a84668ee98ea (12 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
e7b5a672b8d600e5d2b3021d6e1804eb (13 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
753afdbeae4cdd39032285ca635ea51a (14 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
4ea46ce099c17587b064c003280e130a (15 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
90cb0fc165da7cf677b8bfc4ec1850d4 (16 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
9b5e03d5d38166f9be2a8ecc9d393780 (17 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
e9638c105de9dab6a7f889ca0da80227 (18 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
60bfa08feb8d82ceed03b4dc56979d3f (19 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:45 INFO region_mover: Moving region
a37988e554deff2b1a8c7477639f3872 (20 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
135966bd36f58329f532fdd8aa4b0bc7 (21 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
6d2ba19a71695ed8b97eac6f772c37b6 (22 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
8b8804d988d60da967fe5fd1cea44f96 (23 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
a133c11bc0b5e22d30e4d92e40e8adf8 (24 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
18c67cefc95a0484242b74df70160d3d (25 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
9ea96fe31480fededf978d59a9ae148d (26 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
af85ba099ccb1f51630ae4a806918150 (27 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
57a5bbe2c33b13ec6dc4e28c21324b45 (28 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
3a28d3a027385a65bcd001516bc8da5f (29 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:49 INFO region_mover: Moving region
00fd5546feaeb573e7232825d4b2af88 (30 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
9988b51ab86393b007fec59cccb6cfec (31 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
06d051f424d2ac80fc896bad1fc70a37 (32 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
686a74d0e16586abbc6414af79ebbc45 (33 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
b139540fa33e3460375f913550a00403 (34 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
9639ffcd59bfe255313bb504cf3c5c26 (35 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
e9965f1194f422276ed36de3b57fbb7d (36 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
6f4a67e9e8863dbcc3225977bb2b50a3 (37 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
e212ebcaa40162bda4f113e9103dd317 (38 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
0b76c187b57322876972a6a4d33df156 (39 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:52 INFO region_mover: Moving region
c9307492fda9f61f9b1e07b3dfbff137 (40 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
b080758e6ab4bc1c15cec8a7dabeb9bf (41 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
f2d4375225e6d4b0ff95aeb34f2ea952 (42 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
d87ab7c5aa51da57428a3461a7ca8094 (43 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
b07032e2330ecda3716b2ce5b335ac72 (44 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
fa6bddd81e34e641e9303444ee7470b9 (45 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
2ea1a87a76863aca50fe84af7a50253c (46 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
e3e466191856ff9fed3ddad62a77a1e8 (47 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
aa373efd5677f1eeb4ef42a96910fcf8 (48 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
08c3f9bf3af9ecd1bb24d95152f6c9b6 (49 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:55 INFO region_mover: Moving region
164d33459ccf79c42290e126f284f9c6 (50 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
df71b41ab76993b32ba2abbb5ba4d37f (51 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
25b6b7bd6ec050d702ece3b0d819742a (52 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
ebf3475952e14bc60f159a2eaae9a9d6 (53 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
59249ea50e8a8c97fdf9779a05d8388a (54 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
3c89480e1b73cf11c8dc8d23858eea0e (55 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
d0148b1da1c81eda0ff48be4906478a2 (56 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
6f0c16ce6f8c8692f5c3524716d94ecd (57 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
1a2dc3904d34b8300933c9e342e859c3 (58 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
801ef112377eeac002d1bdc9e5f7ee0f (59 of 116) to server=node2,60020,1373895516200
13/07/15 09:38:58 INFO region_mover: Moving region
8a9a80356ff77328109a5ebf0b934afd (60 of 116) to server=node2,60020,1373895516200
{code}
I called with --maxthreads 15 but transfers are done 10 by 10... Which is
strange, because default value is 1.
So I tried to directly called graceful_stop:
time ./bin/graceful_stop.sh --restart --reload --debug --maxthreads 50 buldo
And I also got strange results.
So, I have added some traces on the files and here is the result now:
{code}
13/07/15 10:03:56 INFO region_mover: Moving 86 region(s) from
node6,60020,1373896882598 during this cycle
13/07/15 10:03:56 INFO region_mover: Moving region
7e1feaaff3c1be8e413072b4671245bf (1 of 86) to server=buldo,60020,1373896150259
in thread 0
13/07/15 10:03:56 INFO region_mover: Moving region
8abde383e3636318bb7601625b40163c (2 of 86) to server=node4,60020,1373895267449
in thread 1
13/07/15 10:03:56 INFO region_mover: Moving region
5e04334d7f27ffdbfa4e8f052792b110 (3 of 86) to server=node1,60020,1373895391970
in thread 2
13/07/15 10:03:56 INFO region_mover: Moving region
67512d9f22845ba8ee6de91faa186903 (4 of 86) to server=node2,60020,1373895516200
in thread 3
13/07/15 10:03:56 INFO region_mover: Moving region
c1f031019fd580ce00aba7fe7981829a (5 of 86) to server=node3,60020,1373894767796
in thread 4
13/07/15 10:03:56 INFO region_mover: Moving region
dc88edfc0713ca546c635ac7d4cfbe15 (6 of 86) to server=node5,60020,1373894990003
in thread 5
13/07/15 10:03:59 INFO region_mover: Moving region
c06093c3c8cbcace8fd237de3f961f08 (7 of 86) to server=buldo,60020,1373896150259
in thread 0
13/07/15 10:03:59 INFO region_mover: Moving region
76523c4b4850bf7d17db914ccee13301 (8 of 86) to server=node4,60020,1373895267449
in thread 1
13/07/15 10:03:59 INFO region_mover: Moving region
336df9ec1e4aeeda611c284bc981ac55 (9 of 86) to server=node1,60020,1373895391970
in thread 2
13/07/15 10:03:59 INFO region_mover: Moving region
b2a5614b21802154b86ddc52e9e70c33 (10 of 86) to server=node2,60020,1373895516200
in thread 3
13/07/15 10:03:59 INFO region_mover: Moving region
a4d57785c95465bb19a76a4e545c6f61 (11 of 86) to server=node3,60020,1373894767796
in thread 4
13/07/15 10:03:59 INFO region_mover: Moving region
4de77fcc4e8e87402db9177bff06dae1 (12 of 86) to server=node5,60020,1373894990003
in thread 5
13/07/15 10:04:00 INFO region_mover: Moving region
af39a4ad71dc04b1423d8fd490aa9d81 (13 of 86) to server=buldo,60020,1373896150259
in thread 0
13/07/15 10:04:00 INFO region_mover: Moving region
083f20d0ad282e70dbfddfb4e8c62acd (14 of 86) to server=node4,60020,1373895267449
in thread 1
13/07/15 10:04:00 INFO region_mover: Moving region
90343db01d7723b7a60e875bb67bddb9 (15 of 86) to server=node1,60020,1373895391970
in thread 2
13/07/15 10:04:00 INFO region_mover: Moving region
fab38f5728767a158803b6a1b66ce769 (16 of 86) to server=node2,60020,1373895516200
in thread 3
13/07/15 10:04:00 INFO region_mover: Moving region
ef52da3e8352fbed3c2e5602407fd72e (17 of 86) to server=node3,60020,1373894767796
in thread 4
13/07/15 10:04:00 INFO region_mover: Moving region
a6f4afc58fc58a454dde201d09353418 (18 of 86) to server=node5,60020,1373894990003
in thread 5
{code}
This is correct. I have 7 nodes, one which is going down so I have 1 source and
6 targets, 0 to 5.
And this:
{code}
13/07/15 10:04:36 INFO region_mover: Moving region
c1f031019fd580ce00aba7fe7981829a (1 of 86) to server=node6,60020,1373897072968
in thread 0
13/07/15 10:04:36 INFO region_mover: Moving region
8abde383e3636318bb7601625b40163c (2 of 86) to server=node6,60020,1373897072968
in thread 1
13/07/15 10:04:36 INFO region_mover: Moving region
dc88edfc0713ca546c635ac7d4cfbe15 (3 of 86) to server=node6,60020,1373897072968
in thread 2
13/07/15 10:04:36 INFO region_mover: Moving region
5e04334d7f27ffdbfa4e8f052792b110 (4 of 86) to server=node6,60020,1373897072968
in thread 3
13/07/15 10:04:36 INFO region_mover: Moving region
67512d9f22845ba8ee6de91faa186903 (5 of 86) to server=node6,60020,1373897072968
in thread 4
13/07/15 10:04:36 INFO region_mover: Moving region
7e1feaaff3c1be8e413072b4671245bf (6 of 86) to server=node6,60020,1373897072968
in thread 5
13/07/15 10:04:36 INFO region_mover: Moving region
c06093c3c8cbcace8fd237de3f961f08 (7 of 86) to server=node6,60020,1373897072968
in thread 6
13/07/15 10:04:36 INFO region_mover: Moving region
4de77fcc4e8e87402db9177bff06dae1 (8 of 86) to server=node6,60020,1373897072968
in thread 7
13/07/15 10:04:36 INFO region_mover: Moving region
336df9ec1e4aeeda611c284bc981ac55 (9 of 86) to server=node6,60020,1373897072968
in thread 8
13/07/15 10:04:36 INFO region_mover: Moving region
b2a5614b21802154b86ddc52e9e70c33 (10 of 86) to server=node6,60020,1373897072968
in thread 9
13/07/15 10:04:42 INFO region_mover: Moving region
a4d57785c95465bb19a76a4e545c6f61 (11 of 86) to server=node6,60020,1373897072968
in thread 0
13/07/15 10:04:42 INFO region_mover: Moving region
76523c4b4850bf7d17db914ccee13301 (12 of 86) to server=node6,60020,1373897072968
in thread 1
13/07/15 10:04:42 INFO region_mover: Moving region
af39a4ad71dc04b1423d8fd490aa9d81 (13 of 86) to server=node6,60020,1373897072968
in thread 2
13/07/15 10:04:42 INFO region_mover: Moving region
a6f4afc58fc58a454dde201d09353418 (14 of 86) to server=node6,60020,1373897072968
in thread 3
13/07/15 10:04:42 INFO region_mover: Moving region
083f20d0ad282e70dbfddfb4e8c62acd (15 of 86) to server=node6,60020,1373897072968
in thread 4
13/07/15 10:04:42 INFO region_mover: Moving region
fab38f5728767a158803b6a1b66ce769 (16 of 86) to server=node6,60020,1373897072968
in thread 5
13/07/15 10:04:42 INFO region_mover: Moving region
ef52da3e8352fbed3c2e5602407fd72e (17 of 86) to server=node6,60020,1373897072968
in thread 6
13/07/15 10:04:42 INFO region_mover: Moving region
90343db01d7723b7a60e875bb67bddb9 (18 of 86) to server=node6,60020,1373897072968
in thread 7
13/07/15 10:04:42 INFO region_mover: Moving region
b139540fa33e3460375f913550a00403 (19 of 86) to server=node6,60020,1373897072968
in thread 8
13/07/15 10:04:42 INFO region_mover: Moving region
cc87da68186cfe25cdf7395be9d12f96 (20 of 86) to server=node6,60020,1373897072968
in thread 9
13/07/15 10:04:43 INFO region_mover: Moving region
9639ffcd59bfe255313bb504cf3c5c26 (21 of 86) to server=node6,60020,1373897072968
in thread 0
13/07/15 10:04:43 INFO region_mover: Moving region
d43b99bdb63a0629d9722b491518f684 (22 of 86) to server=node6,60020,1373897072968
in thread 1
13/07/15 10:04:43 INFO region_mover: Moving region
704a1653c2e983f40a2746d1cb7e5c7d (23 of 86) to server=node6,60020,1373897072968
in thread 2
13/07/15 10:04:43 INFO region_mover: Moving region
19d8530b5d619dc88d6ac3a40e13ebc5 (24 of 86) to server=node6,60020,1373897072968
in thread 3
13/07/15 10:04:43 INFO region_mover: Moving region
06d051f424d2ac80fc896bad1fc70a37 (25 of 86) to server=node6,60020,1373897072968
in thread 4
13/07/15 10:04:43 INFO region_mover: Moving region
b080758e6ab4bc1c15cec8a7dabeb9bf (26 of 86) to server=node6,60020,1373897072968
in thread 5
13/07/15 10:04:43 INFO region_mover: Moving region
c9307492fda9f61f9b1e07b3dfbff137 (27 of 86) to server=node6,60020,1373897072968
in thread 6
13/07/15 10:04:43 INFO region_mover: Moving region
686a74d0e16586abbc6414af79ebbc45 (28 of 86) to server=node6,60020,1373897072968
in thread 7
13/07/15 10:04:43 INFO region_mover: Moving region
e212ebcaa40162bda4f113e9103dd317 (29 of 86) to server=node6,60020,1373897072968
in thread 8
13/07/15 10:04:44 INFO region_mover: Moving region
dfcd750be0e2b546d76c752d522c52af (30 of 86) to server=node6,60020,1373897072968
in thread 9
{code}
Which is NOT correct! Threads always stop after 10, however, I called with 50!
Parameter is received correctly:
{code}
hbase@node3:~/hbase-0.94.3$ time ./bin/graceful_stop.sh --restart --reload
--debug --maxthreads 50 node6
Doing rolling with maxthreads=50
Disabling balancer! (if required)
Previous balancer state was true
{code}
So I looked at the code, and the issue is here:
{code}
while thread_index < 10 && counter < regions.length && thread_index <
options[:maxthreads] do
{code}
Which should be
{code}
while counter < regions.length && thread_index < options[:maxthreads] do
{code}
Which this change, regions are assigned back faster:
{code}
node6: starting regionserver, logging to
/home/hbase/hbase-0.94.3/bin/../logs/hbase-hbase-regionserver-node6.out
Reloading node6 region(s)
13/07/15 10:08:41 INFO region_mover: hostname=node6 is not up yet, waiting
13/07/15 10:08:42 INFO region_mover: hostname=node6 is not up yet, waiting
13/07/15 10:08:42 INFO region_mover: Moving 86 regions to
node6,60020,1373897320069
13/07/15 10:08:42 INFO region_mover: Moving region
8abde383e3636318bb7601625b40163c (1 of 86) to server=node6,60020,1373897320069
in thread 0
13/07/15 10:08:42 INFO region_mover: Moving region
5e04334d7f27ffdbfa4e8f052792b110 (2 of 86) to server=node6,60020,1373897320069
in thread 1
13/07/15 10:08:42 INFO region_mover: Moving region
c1f031019fd580ce00aba7fe7981829a (3 of 86) to server=node6,60020,1373897320069
in thread 2
13/07/15 10:08:46 INFO region_mover: Moving region
7e1feaaff3c1be8e413072b4671245bf (4 of 86) to server=node6,60020,1373897320069
in thread 3
13/07/15 10:08:46 INFO region_mover: Moving region
67512d9f22845ba8ee6de91faa186903 (5 of 86) to server=node6,60020,1373897320069
in thread 4
13/07/15 10:08:46 INFO region_mover: Moving region
dc88edfc0713ca546c635ac7d4cfbe15 (6 of 86) to server=node6,60020,1373897320069
in thread 5
13/07/15 10:08:46 INFO region_mover: Moving region
a4d57785c95465bb19a76a4e545c6f61 (7 of 86) to server=node6,60020,1373897320069
in thread 6
13/07/15 10:08:46 INFO region_mover: Moving region
b2a5614b21802154b86ddc52e9e70c33 (8 of 86) to server=node6,60020,1373897320069
in thread 7
13/07/15 10:08:46 INFO region_mover: Moving region
4de77fcc4e8e87402db9177bff06dae1 (9 of 86) to server=node6,60020,1373897320069
in thread 8
13/07/15 10:08:46 INFO region_mover: Moving region
c06093c3c8cbcace8fd237de3f961f08 (10 of 86) to server=node6,60020,1373897320069
in thread 9
13/07/15 10:08:46 INFO region_mover: Moving region
76523c4b4850bf7d17db914ccee13301 (11 of 86) to server=node6,60020,1373897320069
in thread 10
13/07/15 10:08:46 INFO region_mover: Moving region
336df9ec1e4aeeda611c284bc981ac55 (12 of 86) to server=node6,60020,1373897320069
in thread 11
13/07/15 10:08:46 INFO region_mover: Moving region
af39a4ad71dc04b1423d8fd490aa9d81 (13 of 86) to server=node6,60020,1373897320069
in thread 12
13/07/15 10:08:46 INFO region_mover: Moving region
fab38f5728767a158803b6a1b66ce769 (14 of 86) to server=node6,60020,1373897320069
in thread 13
13/07/15 10:08:46 INFO region_mover: Moving region
ef52da3e8352fbed3c2e5602407fd72e (15 of 86) to server=node6,60020,1373897320069
in thread 14
13/07/15 10:08:46 INFO region_mover: Moving region
a6f4afc58fc58a454dde201d09353418 (16 of 86) to server=node6,60020,1373897320069
in thread 15
13/07/15 10:08:46 INFO region_mover: Moving region
083f20d0ad282e70dbfddfb4e8c62acd (17 of 86) to server=node6,60020,1373897320069
in thread 16
13/07/15 10:08:46 INFO region_mover: Moving region
90343db01d7723b7a60e875bb67bddb9 (18 of 86) to server=node6,60020,1373897320069
in thread 17
13/07/15 10:08:46 INFO region_mover: Moving region
b139540fa33e3460375f913550a00403 (19 of 86) to server=node6,60020,1373897320069
in thread 18
13/07/15 10:08:46 INFO region_mover: Moving region
d43b99bdb63a0629d9722b491518f684 (20 of 86) to server=node6,60020,1373897320069
in thread 19
13/07/15 10:08:46 INFO region_mover: Moving region
9639ffcd59bfe255313bb504cf3c5c26 (21 of 86) to server=node6,60020,1373897320069
in thread 20
13/07/15 10:08:46 INFO region_mover: Moving region
cc87da68186cfe25cdf7395be9d12f96 (22 of 86) to server=node6,60020,1373897320069
in thread 21
13/07/15 10:08:46 INFO region_mover: Moving region
704a1653c2e983f40a2746d1cb7e5c7d (23 of 86) to server=node6,60020,1373897320069
in thread 22
13/07/15 10:08:46 INFO region_mover: Moving region
19d8530b5d619dc88d6ac3a40e13ebc5 (24 of 86) to server=node6,60020,1373897320069
in thread 23
13/07/15 10:08:47 INFO region_mover: Moving region
b080758e6ab4bc1c15cec8a7dabeb9bf (25 of 86) to server=node6,60020,1373897320069
in thread 24
13/07/15 10:08:47 INFO region_mover: Moving region
06d051f424d2ac80fc896bad1fc70a37 (26 of 86) to server=node6,60020,1373897320069
in thread 25
13/07/15 10:08:47 INFO region_mover: Moving region
c9307492fda9f61f9b1e07b3dfbff137 (27 of 86) to server=node6,60020,1373897320069
in thread 26
13/07/15 10:08:47 INFO region_mover: Moving region
686a74d0e16586abbc6414af79ebbc45 (28 of 86) to server=node6,60020,1373897320069
in thread 27
13/07/15 10:08:47 INFO region_mover: Moving region
dfcd750be0e2b546d76c752d522c52af (29 of 86) to server=node6,60020,1373897320069
in thread 28
13/07/15 10:08:47 INFO region_mover: Moving region
e212ebcaa40162bda4f113e9103dd317 (30 of 86) to server=node6,60020,1373897320069
in thread 29
13/07/15 10:08:47 INFO region_mover: Moving region
850b52020a3c58465348bc7019cb18d5 (31 of 86) to server=node6,60020,1373897320069
in thread 30
13/07/15 10:08:47 INFO region_mover: Moving region
d87ab7c5aa51da57428a3461a7ca8094 (32 of 86) to server=node6,60020,1373897320069
in thread 31
13/07/15 10:08:47 INFO region_mover: Moving region
84fb7155803a865a0cfd667aebb1319e (33 of 86) to server=node6,60020,1373897320069
in thread 32
13/07/15 10:08:47 INFO region_mover: Moving region
93f92112f0a7f0d710cfa4f3bb3969f2 (34 of 86) to server=node6,60020,1373897320069
in thread 33
13/07/15 10:08:47 INFO region_mover: Moving region
1b1dbb16623f848f864039c835cc762c (35 of 86) to server=node6,60020,1373897320069
in thread 34
13/07/15 10:08:47 INFO region_mover: Moving region
a545b4c68c34b113a7aaa6adaffdd6a7 (36 of 86) to server=node6,60020,1373897320069
in thread 35
13/07/15 10:08:47 INFO region_mover: Moving region
35d4440e947a9fad6afb185db7e2c5f8 (37 of 86) to server=node6,60020,1373897320069
in thread 36
13/07/15 10:08:47 INFO region_mover: Moving region
328f2c345f1c68cf231c564c81fb7602 (38 of 86) to server=node6,60020,1373897320069
in thread 37
13/07/15 10:08:47 INFO region_mover: Moving region
1a561226889cc4a1d022b9055c86ac88 (39 of 86) to server=node6,60020,1373897320069
in thread 38
13/07/15 10:08:47 INFO region_mover: Moving region
995a13fc260782460eda330ab905c1ae (40 of 86) to server=node6,60020,1373897320069
in thread 39
13/07/15 10:08:47 INFO region_mover: Moving region
be63b5b8804e62e15fb484426892dc2a (41 of 86) to server=node6,60020,1373897320069
in thread 40
13/07/15 10:08:47 INFO region_mover: Moving region
33d846ea97d89df08c06fed5315f1a4e (42 of 86) to server=node6,60020,1373897320069
in thread 41
13/07/15 10:08:47 INFO region_mover: Moving region
b42df270d6b1c81ec0f464b38d554caa (43 of 86) to server=node6,60020,1373897320069
in thread 42
13/07/15 10:08:47 INFO region_mover: Moving region
27516238bf5c278bef833f9f8ab578a4 (44 of 86) to server=node6,60020,1373897320069
in thread 43
13/07/15 10:08:47 INFO region_mover: Moving region
390e346b871aae673a5771118da3e218 (45 of 86) to server=node6,60020,1373897320069
in thread 44
13/07/15 10:08:47 INFO region_mover: Moving region
12ad4888d6a6c5b706ad31eef7075522 (46 of 86) to server=node6,60020,1373897320069
in thread 45
13/07/15 10:08:47 INFO region_mover: Moving region
b97a8f3dc029ad6f7a074663ebcb20ae (47 of 86) to server=node6,60020,1373897320069
in thread 46
13/07/15 10:08:47 INFO region_mover: Moving region
87c2a872fd747934bfadc2d8baf23cc1 (48 of 86) to server=node6,60020,1373897320069
in thread 47
13/07/15 10:08:47 INFO region_mover: Moving region
f6c158948af66e84b6ac50bb2d06328b (49 of 86) to server=node6,60020,1373897320069
in thread 48
13/07/15 10:08:47 INFO region_mover: Moving region
3a43e4cce05e390c5c2a0a3c7de27609 (50 of 86) to server=node6,60020,1373897320069
in thread 49
13/07/15 10:08:53 INFO region_mover: Moving region
97fea8530a248e90dd1fc702cf9f5615 (51 of 86) to server=node6,60020,1373897320069
in thread 0
13/07/15 10:08:53 INFO region_mover: Moving region
e1db3e40fca7b65d21f89ea299b7a6ab (52 of 86) to server=node6,60020,1373897320069
in thread 1
13/07/15 10:08:53 INFO region_mover: Moving region
080bbb7863d7ea557518349f4aaec6f2 (53 of 86) to server=node6,60020,1373897320069
in thread 2
13/07/15 10:08:53 INFO region_mover: Moving region
9aae66c0345ab31600dd19b3846953af (54 of 86) to server=node6,60020,1373897320069
in thread 3
13/07/15 10:08:53 INFO region_mover: Moving region
7560d31b45659d6428aea45341fb50a0 (55 of 86) to server=node6,60020,1373897320069
in thread 4
13/07/15 10:08:53 INFO region_mover: Moving region
479226a40a0eb5917a3022a26b8433f2 (56 of 86) to server=node6,60020,1373897320069
in thread 5
13/07/15 10:08:53 INFO region_mover: Moving region
67b53c3f300459588ada5a03708ddd6a (57 of 86) to server=node6,60020,1373897320069
in thread 6
13/07/15 10:08:53 INFO region_mover: Moving region
dd51d21b7393a9e55058b637a127e972 (58 of 86) to server=node6,60020,1373897320069
in thread 7
13/07/15 10:08:53 INFO region_mover: Moving region
e0d9a50c271e20ea198b651ba6a51506 (59 of 86) to server=node6,60020,1373897320069
in thread 8
13/07/15 10:08:53 INFO region_mover: Moving region
5efeb8e3c9c1b9c36871f8a01546d37f (60 of 86) to server=node6,60020,1373897320069
in thread 9
13/07/15 10:08:53 INFO region_mover: Moving region
6c1bd871810fd82112fa1b6d59e54f20 (61 of 86) to server=node6,60020,1373897320069
in thread 10
13/07/15 10:08:53 INFO region_mover: Moving region
a65a5d0b011b16d6bfb9c255b660488c (62 of 86) to server=node6,60020,1373897320069
in thread 11
13/07/15 10:08:53 INFO region_mover: Moving region
d4353fe99c08edb8afd99f678370f25c (63 of 86) to server=node6,60020,1373897320069
in thread 12
13/07/15 10:08:53 INFO region_mover: Moving region
8b77e785125f6266d5e85d5e6dffc7b6 (64 of 86) to server=node6,60020,1373897320069
in thread 13
13/07/15 10:08:56 INFO region_mover: Moving region
fe8ae36515c4ff12d7c789dec5561255 (65 of 86) to server=node6,60020,1373897320069
in thread 14
13/07/15 10:08:56 INFO region_mover: Moving region
3fda0f5c7a845ea17c69d0c0a798011c (66 of 86) to server=node6,60020,1373897320069
in thread 15
13/07/15 10:08:56 INFO region_mover: Moving region
2d96d34b345147bb2a1f4ce9093e8ac1 (67 of 86) to server=node6,60020,1373897320069
in thread 16
13/07/15 10:08:56 INFO region_mover: Moving region
90d6c08774576abaec3769402084dcd1 (68 of 86) to server=node6,60020,1373897320069
in thread 17
13/07/15 10:08:56 INFO region_mover: Moving region
6fb8972955a0ef3d7aa3b92eaedc6e7e (69 of 86) to server=node6,60020,1373897320069
in thread 18
13/07/15 10:08:56 INFO region_mover: Moving region
2f7d30310466c1933263f03fda043dea (70 of 86) to server=node6,60020,1373897320069
in thread 19
13/07/15 10:08:56 INFO region_mover: Moving region
ea4f03a2c30f28f0ea00c40a1255d26a (71 of 86) to server=node6,60020,1373897320069
in thread 20
13/07/15 10:08:56 INFO region_mover: Moving region
b6aa2ce729d983cba4cb4032d6844340 (72 of 86) to server=node6,60020,1373897320069
in thread 21
13/07/15 10:08:56 INFO region_mover: Moving region
9913a2995938209dd92bd88c26896bf5 (73 of 86) to server=node6,60020,1373897320069
in thread 22
13/07/15 10:08:56 INFO region_mover: Moving region
d397d88e1382b423e10d1e8f6fa7373c (74 of 86) to server=node6,60020,1373897320069
in thread 23
13/07/15 10:08:56 INFO region_mover: Moving region
c73742d4ee051408f4ba046f2a0d8d91 (75 of 86) to server=node6,60020,1373897320069
in thread 24
13/07/15 10:08:56 INFO region_mover: Moving region
5907da5fd675f4b37a536a8e6c4fe707 (76 of 86) to server=node6,60020,1373897320069
in thread 25
13/07/15 10:08:56 INFO region_mover: Moving region
1557dbe459c1fb8aefd5a20cab3ff48c (77 of 86) to server=node6,60020,1373897320069
in thread 26
13/07/15 10:08:56 INFO region_mover: Moving region
87e54006d207cab2b4e2e4eca6d28970 (78 of 86) to server=node6,60020,1373897320069
in thread 27
13/07/15 10:08:56 INFO region_mover: Moving region
d78195fe223c3e66a5ac88b20dc44171 (79 of 86) to server=node6,60020,1373897320069
in thread 28
13/07/15 10:08:56 INFO region_mover: Moving region
6c8774f2998956a65eadb6a60a53e989 (80 of 86) to server=node6,60020,1373897320069
in thread 29
13/07/15 10:08:56 INFO region_mover: Moving region
96eb5def7988120bce6276f4d77565db (81 of 86) to server=node6,60020,1373897320069
in thread 30
13/07/15 10:08:56 INFO region_mover: Moving region
5e7ecf6fc95285551038bfe6e9b8f1f7 (82 of 86) to server=node6,60020,1373897320069
in thread 31
13/07/15 10:08:56 INFO region_mover: Moving region
39417a882f11daf537d0060cb7a21317 (83 of 86) to server=node6,60020,1373897320069
in thread 32
13/07/15 10:08:56 INFO region_mover: Moving region
8703330d16c09882c5d0729257e2c2a8 (84 of 86) to server=node6,60020,1373897320069
in thread 33
13/07/15 10:08:56 INFO region_mover: Moving region
bbc7c34c89527ae1a15e4ce763ec93e3 (85 of 86) to server=node6,60020,1373897320069
in thread 34
13/07/15 10:08:56 INFO region_mover: Moving region
1d047071a5b9074e0d25de9e71872ed6 (86 of 86) to server=node6,60020,1373897320069
in thread 35
Reloaded node6 region(s)
Restoring balancer state to true
{code}
And it's now WAY faster! With maxthreads 50 it takes 16 minutes now instead of
about 40 minutes before ;)
Can you apply this modification in your last patch?
> region_mover.rb should move multiple regions at a time
> ------------------------------------------------------
>
> Key: HBASE-8803
> URL: https://issues.apache.org/jira/browse/HBASE-8803
> Project: HBase
> Issue Type: Bug
> Components: Usability
> Affects Versions: 0.98.0, 0.94.8, 0.95.1
> Reporter: Jean-Marc Spaggiari
> Assignee: Jean-Marc Spaggiari
> Fix For: 0.98.0, 0.95.2, 0.94.10
>
> Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch,
> HBASE-8803-v1-trunk.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When there is many regions in a cluster, rolling_restart can take hours
> because region_mover is moving the regions one by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
