[
https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711694#comment-13711694
]
Nick Dimiduk commented on HBASE-8803:
-------------------------------------
bq. Are you sure you are using the patched version?
Definitely running v4. Looks like this patch includes the thread count in the
load command, but not in unload.
bq. also, how many active nodes do you have in you HBase cluster?
This cluster has 9 RS's, so your logic makes sense. It looks like unloading
regions is still operating 8 at a time for me.
{noformat}
Unloading hor18n39.gq1.ygridcore.net region(s)
...
Valid region move targets:
[392/465]
hor18n37.gq1.ygridcore.net,60020,1374094975268
hor17n37.gq1.ygridcore.net,60020,1374094975264
hor18n35.gq1.ygridcore.net,60020,1374094975327
hor17n39.gq1.ygridcore.net,60020,1374094975281
hor18n36.gq1.ygridcore.net,60020,1374094975254
hor17n36.gq1.ygridcore.net,60020,1374094975277
hor17n34.gq1.ygridcore.net,60020,1374094975291
hor18n38.gq1.ygridcore.net,60020,1374094975259
13/07/17 21:44:38 INFO region_mover: Moving 330 region(s) from
hor18n39.gq1.ygridcore.net,60020,1374094975326 during this cycle
13/07/17 21:44:38 INFO region_mover: Moving region
b59050cf97aabcef838e3c50e93e6d13 (1 of 330) to
server=hor18n37.gq1.ygridcore.net,60020,1374094975268
13/07/17 21:44:38 INFO region_mover: Moving region
1f1a21f370963529326e1e8647c77977 (2 of 330) to
server=hor17n37.gq1.ygridcore.net,60020,1374094975264
13/07/17 21:44:38 INFO region_mover: Moving region
214a5b6dd53cb94e5fe549712df10918 (3 of 330) to
server=hor18n35.gq1.ygridcore.net,60020,1374094975327
13/07/17 21:44:38 INFO region_mover: Moving region
157d5afaffe570ef1d5535162ee61a73 (4 of 330) to
server=hor17n39.gq1.ygridcore.net,60020,1374094975281
13/07/17 21:44:38 INFO region_mover: Moving region
af5448c90e78a8f0d935efb0b380502e (5 of 330) to
server=hor18n36.gq1.ygridcore.net,60020,1374094975254
13/07/17 21:44:38 INFO region_mover: Moving region
bbe7e56f1e895379bfb6bbc2ef3ebec7 (6 of 330) to
server=hor17n36.gq1.ygridcore.net,60020,1374094975277
13/07/17 21:44:38 INFO region_mover: Moving region
cdf96aa8d79255991c82267052cf9870 (7 of 330) to
server=hor17n34.gq1.ygridcore.net,60020,1374094975291
13/07/17 21:44:38 INFO region_mover: Moving region
7d0a02f452c334a12026b45346a87d36 (8 of 330) to
server=hor18n38.gq1.ygridcore.net,60020,1374094975259
13/07/17 21:45:41 INFO region_mover: Moving region
01408a1987f7a0cb7fb657a19bddc2d5 (9 of 330) to
server=hor18n37.gq1.ygridcore.net,60020,1374094975268
13/07/17 21:45:41 INFO region_mover: Moving region
45d7cf6b5997310b022f315e52f64da2 (10 of 330) to
server=hor17n37.gq1.ygridcore.net,60020,1374094975264
13/07/17 21:45:41 INFO region_mover: Moving region
353a09c07053144a598be04a53afa82e (11 of 330) to
server=hor18n35.gq1.ygridcore.net,60020,1374094975327
13/07/17 21:45:41 INFO region_mover: Moving region
5f4e8165d58c0672c8c6e58ddbff486c (12 of 330) to
server=hor17n39.gq1.ygridcore.net,60020,1374094975281
13/07/17 21:45:41 INFO region_mover: Moving region
234265b7ec4e535cecafcdd83f09dab7 (13 of 330) to
server=hor18n36.gq1.ygridcore.net,60020,1374094975254
13/07/17 21:45:41 INFO region_mover: Moving region
9f54ff65021922f7c13f4372dbcb8f98 (14 of 330) to
server=hor17n36.gq1.ygridcore.net,60020,1374094975277
13/07/17 21:45:41 INFO region_mover: Moving region
0db22200ccea02196dfea00b33811413 (15 of 330) to
server=hor17n34.gq1.ygridcore.net,60020,1374094975291
13/07/17 21:45:41 INFO region_mover: Moving region
905210e29ef683ecf78f01050c4b1148 (16 of 330) to
server=hor18n38.gq1.ygridcore.net,60020,1374094975259
...
{noformat}
Loading regions is launching threads, at least according to your new log
messages. Here I would expect 30 threads to spin up all at once. Instead, I see
one at a time:
{noformat}
Reloading hor18n39.gq1.ygridcore.net region(s)
...
13/07/17 21:54:27 INFO region_mover: Moving 330 regions to
hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:56:47 INFO region_mover: Moving region
7d0a02f452c334a12026b45346a87d36 (1 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 0
13/07/17 21:56:54 INFO region_mover: Moving region
af5448c90e78a8f0d935efb0b380502e (2 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 1
13/07/17 21:57:01 INFO region_mover: Moving region
157d5afaffe570ef1d5535162ee61a73 (3 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 2
13/07/17 21:57:06 INFO region_mover: Moving region
cdf96aa8d79255991c82267052cf9870 (4 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 3
13/07/17 21:57:12 INFO region_mover: Moving region
214a5b6dd53cb94e5fe549712df10918 (5 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 4
13/07/17 21:57:18 INFO region_mover: Moving region
b59050cf97aabcef838e3c50e93e6d13 (6 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 5
13/07/17 21:57:23 INFO region_mover: Moving region
bbe7e56f1e895379bfb6bbc2ef3ebec7 (7 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 6
13/07/17 21:57:28 INFO region_mover: Moving region
1f1a21f370963529326e1e8647c77977 (8 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 7
13/07/17 21:57:33 INFO region_mover: Moving region
9f54ff65021922f7c13f4372dbcb8f98 (9 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 8
13/07/17 21:57:37 INFO region_mover: Moving region
905210e29ef683ecf78f01050c4b1148 (10 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 9
13/07/17 21:57:42 INFO region_mover: Moving region
234265b7ec4e535cecafcdd83f09dab7 (11 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 10
...
13/07/17 21:59:17 INFO region_mover: Moving region
da0f100db394d18e5b6311938097ec3e (35 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 28
13/07/17 21:59:20 INFO region_mover: Moving region
900effc2e95064173fc0a3fab047b119 (36 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 29
13/07/17 21:59:28 INFO region_mover: Moving region
1b78694f962d7c315e4f6b64f07af4aa (37 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 0
13/07/17 21:59:32 INFO region_mover: Moving region
3e47c04afce79818e5ead7d8f46e7128 (38 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 1
13/07/17 21:59:36 INFO region_mover: Moving region
7cce3224f05e438f0e011d6f26b738c0 (39 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 2
13/07/17 21:59:39 INFO region_mover: Region
t1,1735ed70,1373581705674.ef04ef42d7eb7d7e472671661cc48ee0. (40 of 330) already
on target server=hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:59:43 INFO region_mover: Region
t1,2888871d,1373581705680.483e522559ecd6b0fdd96dbd6b593c7d. (41 of 330) already
on target server=hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:59:46 INFO region_mover: Moving region
6337b0857ccc6d995082582841e467d7 (42 of 330) to
server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 3
...
{noformat}
> region_mover.rb should move multiple regions at a time
> ------------------------------------------------------
>
> Key: HBASE-8803
> URL: https://issues.apache.org/jira/browse/HBASE-8803
> Project: HBase
> Issue Type: Bug
> Components: Usability
> Affects Versions: 0.98.0, 0.94.8, 0.95.1
> Reporter: Jean-Marc Spaggiari
> Assignee: Jean-Marc Spaggiari
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch,
> HBASE-8803-v1-trunk.patch, HBASE-8803-v2-0.94.patch,
> HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, HBASE-8803-v4-0.94.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When there is many regions in a cluster, rolling_restart can take hours
> because region_mover is moving the regions one by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
