[
https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722546#comment-13722546
]
Jean-Marc Spaggiari commented on HBASE-8803:
--------------------------------------------
Hi Nick,
Can you add the thread trace in the unload log and retry? I tried again on my
side and it's still correct.
Also, I verified that regions are moved only once:
{code}
hbase@node3:~/hbase-0.94$ cat logs/hbase-hbase-master-node3.log | grep
dd86a04810f2839ee7842733b7a6ec78
2013-07-29 07:59:33,238 INFO org.apache.hadoop.hbase.master.HMaster: Added move
plan
hri=crc,\x00\x00\x00\x00+\xB1\x108seattle.cbsradio.stats.com,1365475884535.dd86a04810f2839ee7842733b7a6ec78.,
src=node3,60020,1375017424052, dest=node6,60020,1375016625878, running balancer
2013-07-29 08:01:40,131 INFO org.apache.hadoop.hbase.master.HMaster: Added move
plan
hricrc,\x00\x00\x00\x00+\xB1\x108seattle.cbsradio.stats.com,1365475884535.dd86a04810f2839ee7842733b7a6ec78.,
src=node6,60020,1375016625878, dest=node3,60020,1375099217169, running balancer
{code}
Let me know if you need an updated patch for the unload output. What you should
see is something like that:
{code}
Gracefully restarting: node1
Doing rolling with maxthreads=30
Disabling balancer! (if required)
Previous balancer state was true
Unloading node1 region(s)
Valid region move targets:
node4,60020,1375016907951
buldo,60020,1375016770574
node5,60020,1375017021737
node7,60020,1375017151389
node2,60020,1375015919731
node3,60020,1375015482026
node6,60020,1375016625878
13/07/28 09:13:31 INFO region_mover: Moving 88 region(s) from
node1,60020,1375015859230 during this cycle
13/07/28 09:13:31 INFO region_mover: Moving region
0543a51c262cf78561e4d2d07867385b (1 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:31 INFO region_mover: Moving region
86fe96d85203f15ec769a1ab84ea652e (2 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:31 INFO region_mover: Moving region
323b9de45db8b4bc8b200a33550bf339 (3 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:31 INFO region_mover: Moving region
796665161dc582ac0beb37fdcdd82db8 (4 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:31 INFO region_mover: Moving region
a5d61d600d21fb05253e8afa2489d7ef (5 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:31 INFO region_mover: Moving region
3d326f914d8c3eb40db8d48e945bd618 (6 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:31 INFO region_mover: Moving region
1c65891f0a76bc26b45364cf2311841e (7 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:32 INFO region_mover: Moving region
bd4bbdcc078742d0d9474bcfc91cbb3b (8 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:32 INFO region_mover: Moving region
5613e945414d1be0e0eaff89891554f1 (9 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:32 INFO region_mover: Moving region
fe44b401dfe7e4924601ef909032ad28 (10 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:32 INFO region_mover: Moving region
0d4762ae7f855f3ff14abdf4c7e1acec (11 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:32 INFO region_mover: Moving region
86e0f44b1ca082fada429ed2ee798806 (12 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:32 INFO region_mover: Moving region
8d295e0ceb2bd62952d9e47dd81699c6 (13 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:32 INFO region_mover: Moving region
ae0b4649917574b82f7d8f2779800cd0 (14 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:34 INFO region_mover: Moving region
73a6401006f8e7e1a7c2d5e7d9e4436d (15 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:34 INFO region_mover: Moving region
8beeb42f41b04aa38c020e2173e4ed5e (16 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:34 INFO region_mover: Moving region
47ef6f201b2b96558dafd50f6062c1f3 (17 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:34 INFO region_mover: Moving region
4c6e30b8d0d320be240d4edeffbfff17 (18 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:34 INFO region_mover: Moving region
09d98ca982d2c36f616ec568deafc954 (19 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:34 INFO region_mover: Moving region
39bcc1fc090c8a939dca5b45e2119122 (20 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:34 INFO region_mover: Moving region
3ded89d9383bdd518a5d24d7c478ed50 (21 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:36 INFO region_mover: Moving region
05a6717f7a01222017c5328cc6cf3401 (22 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:36 INFO region_mover: Moving region
08cfbeed57512350323e9172b28157b2 (23 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:36 INFO region_mover: Moving region
2be8650e51a90f6b95d354a6387654d8 (24 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:36 INFO region_mover: Moving region
d4f68fb6560cd1ca0c89c931128eeb60 (25 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:36 INFO region_mover: Moving region
c36c0f3f5dc8081ce23cb446cc6bada4 (26 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:36 INFO region_mover: Moving region
75221b26d619649af5b195dbeabb5703 (27 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:36 INFO region_mover: Moving region
f324f85f12b9669c662ed48d204ee05f (28 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:37 INFO region_mover: Moving region
4e4ccef8c54ff1753153eaa3ba09a2f5 (29 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:37 INFO region_mover: Moving region
5158d360baa4bf264f153ee641591444 (30 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:37 INFO region_mover: Moving region
3598b8e6ff505a0b73dbbcdd92fbfe05 (31 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:37 INFO region_mover: Moving region
15f2c0a63c2f4263d872e993044216bd (32 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:37 INFO region_mover: Moving region
1b0582151f5535dd575ff081576a72ec (33 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:37 INFO region_mover: Moving region
f4ca37fd05fe8beb67295290ad3cec49 (34 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:37 INFO region_mover: Moving region
a8f1ab17de0d112eebd1555f5d830d7a (35 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:39 INFO region_mover: Moving region
83f9e939d38a056769bbc6549ad4f5e3 (36 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:39 INFO region_mover: Moving region
ae4d3e8377165ea98018ed6f6bbcaa52 (37 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:39 INFO region_mover: Moving region
6561c0877b1c3b47d40f6e1095e70f50 (38 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:39 INFO region_mover: Moving region
8c4ef570843987cfcf87a9a0ff07cdf0 (39 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:39 INFO region_mover: Moving region
cff814ae590359d23272e971770a7b87 (40 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:39 INFO region_mover: Moving region
00c619b5b1833be9078b40f759fca9aa (41 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:39 INFO region_mover: Moving region
7e5f13d4041fbc762dd45dc38c497220 (42 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:40 INFO region_mover: Moving region
38531199c8bea5091e52daf52ec5e18d (43 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:40 INFO region_mover: Moving region
85b42c278834ea4720840e44a8b001fc (44 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:40 INFO region_mover: Moving region
c82d62ff4be237568bcf26a3c76ea895 (45 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:40 INFO region_mover: Moving region
044ab9af47b5b4ba4e59ed60a571d0ec (46 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:40 INFO region_mover: Moving region
b2886bc7c5bf915e992fa35a5dc1307c (47 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:40 INFO region_mover: Moving region
13be3a6a55570fc32eab44edadcd4bf0 (48 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:40 INFO region_mover: Moving region
1addfaadcb399b78c5912e40e78f39d7 (49 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:41 INFO region_mover: Moving region
562366f4a5385fb2dc8eb82970da580e (50 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:41 INFO region_mover: Moving region
acd89f0ccd337bc5a6314b1b5bebf258 (51 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:41 INFO region_mover: Moving region
0e3a885b55ea00132e661fa8d12703c9 (52 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:41 INFO region_mover: Moving region
a304b794f0778e74dd577756070ba162 (53 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:41 INFO region_mover: Moving region
98b4b71f9cc3818b9755006e2c04712e (54 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:41 INFO region_mover: Moving region
f57616d308d9b3a9e28b99bf09f746e1 (55 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:41 INFO region_mover: Moving region
8708b53a9455da07ea347ed2720ca7e9 (56 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:42 INFO region_mover: Moving region
fec6841875480f334341f12d91fa55c7 (57 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:42 INFO region_mover: Moving region
700f6ed359b973c2e9f66bc01fa8b00f (58 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:42 INFO region_mover: Moving region
e534663fcd52e358ed389d7a0b186377 (59 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:42 INFO region_mover: Moving region
7ca941fd1996e5b440fb567c88201573 (60 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:42 INFO region_mover: Moving region
a5d3b2f7ce836839d75a8b6263071742 (61 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:42 INFO region_mover: Moving region
8c1b0331c61c14a3d91fe01edf4a22b2 (62 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:42 INFO region_mover: Moving region
de53006a0b87a2a4aefa02780b7d148a (63 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:43 INFO region_mover: Moving region
3304e33f985424141181d9bb56a14457 (64 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:43 INFO region_mover: Moving region
ed0d8d398ed37ff5de73daff21c49c6e (65 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:43 INFO region_mover: Moving region
fff3c06d30c769092d7b287fa0677675 (66 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:43 INFO region_mover: Moving region
e1a32594f393216c7686ec7d1566b3b3 (67 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:43 INFO region_mover: Moving region
8e05d2c6a02bfa43d9a3e18c21da14b6 (68 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:43 INFO region_mover: Moving region
44732a5e862c3a92c6e0f6da4f3caedc (69 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:43 INFO region_mover: Moving region
77ea3447e83a5d60ad944f59606fa1fb (70 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:45 INFO region_mover: Moving region
24a27a676428e16475952232129589a8 (71 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:45 INFO region_mover: Moving region
f4a0228b97983d81fb53a75b14ff87eb (72 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:45 INFO region_mover: Moving region
66f07ed9a93d7b1701f4dbcf1eb96e93 (73 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:45 INFO region_mover: Moving region
80e1f501dc29a0e2efc78e9ebb55d159 (74 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:45 INFO region_mover: Moving region
14bbc4273058426ffc6bcd741e0961bc (75 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:45 INFO region_mover: Moving region
53dfd1db4af6b7f9e2ba30fa9665aa94 (76 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:45 INFO region_mover: Moving region
56629b3eedc81e5d66a2da9c0c388227 (77 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:46 INFO region_mover: Moving region
e26fd44dfbd9b71e98ef40e16bc88cf5 (78 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:46 INFO region_mover: Moving region
3378921a54749b9d952239b0b11953b9 (79 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:46 INFO region_mover: Moving region
a87d5dbe2a67aba7fb6504e416b4c2d5 (80 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:46 INFO region_mover: Moving region
c8b6de6c951a076d0bdb131f342435f7 (81 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:46 INFO region_mover: Moving region
8963fa279eff47da3f49ba1c332444c0 (82 of 88) to server=node2,60020,1375015919731
in thread 4
13/07/28 09:13:46 INFO region_mover: Moving region
dc87f6b873bfa071396175623a23ff6c (83 of 88) to server=node3,60020,1375015482026
in thread 5
13/07/28 09:13:46 INFO region_mover: Moving region
180421ace2ec1b7d7d244403bd76031d (84 of 88) to server=node6,60020,1375016625878
in thread 6
13/07/28 09:13:48 INFO region_mover: Moving region
678b9028651a84c27c4ecebad4819bc9 (85 of 88) to server=node4,60020,1375016907951
in thread 0
13/07/28 09:13:48 INFO region_mover: Moving region
9ff9cfdd527a273309763f0c62e6807c (86 of 88) to server=buldo,60020,1375016770574
in thread 1
13/07/28 09:13:48 INFO region_mover: Moving region
0e01b4178ec53500bc1af096db8bb8bc (87 of 88) to server=node5,60020,1375017021737
in thread 2
13/07/28 09:13:48 INFO region_mover: Moving region
d3483133858ec6c12989f160cf861b9e (88 of 88) to server=node7,60020,1375017151389
in thread 3
13/07/28 09:13:48 INFO region_mover: Wrote list of moved regions to /tmp/node1
Unloaded node1 region(s)
node1: stopping regionserver..
node1: starting regionserver, logging to
/home/hbase/hbase-0.94.3/bin/../logs/hbase-hbase-regionserver-node1.out
{code}
> region_mover.rb should move multiple regions at a time
> ------------------------------------------------------
>
> Key: HBASE-8803
> URL: https://issues.apache.org/jira/browse/HBASE-8803
> Project: HBase
> Issue Type: Bug
> Components: Usability
> Affects Versions: 0.98.0, 0.94.8, 0.95.1
> Reporter: Jean-Marc Spaggiari
> Assignee: Jean-Marc Spaggiari
> Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch,
> HBASE-8803-v1-trunk.patch, HBASE-8803-v2-0.94.patch,
> HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, HBASE-8803-v4-0.94.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When there is many regions in a cluster, rolling_restart can take hours
> because region_mover is moving the regions one by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
