[jira] [Updated] (HBASE-10632) Region lost in limbo after ArrayIndexOutOfBoundsException during assignment

2014-03-03 Thread Enis Soztutar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-10632:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I've committed this to trunk, 0.98, 0.96 and hbase-10070 branches. Thanks for 
reviews. 

> Region lost in limbo after ArrayIndexOutOfBoundsException during assignment
> ---
>
> Key: HBASE-10632
> URL: https://issues.apache.org/jira/browse/HBASE-10632
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: hbase-10070
>Reporter: Nick Dimiduk
>Assignee: Enis Soztutar
> Fix For: 0.96.2, 0.98.1, 0.99.0, hbase-10070
>
> Attachments: hbase-10632_v1.patch
>
>
> Discovered while running IntegrationTestBigLinkedList. Region 
> 24d68aa7239824e42390a77b7212fcbf is scheduled for move from hor13n19 to 
> hor13n13. During the process an exception is thrown.
> {noformat}
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioning {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} will be handled by SSH 
> for hor13n19.gq1.ygridcore.net,60020,1393341563552
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning 7 region(s) that 
> hor13n19.gq1.ygridcore.net,60020,1393341563552 was carrying (and 0 regions(s) 
> that were opening on this server)
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning region with rs = 
> {24d68aa7239824e42390a77b7212fcbf state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} and deleting zk node 
> if exists
> 2014-02-25 15:30:42,623 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioned {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} to 
> {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> 2014-02-25 15:30:42,623 DEBUG [AM.ZK.Worker-pool2-t46] 
> master.AssignmentManager: Znode 
> IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.
>  deleted, state: {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, 
> ts=1393342242623, server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:30:43,993 ERROR [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> executor.EventHandler: Caught throwable while processing event 
> M_SERVER_SHUTDOWN
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:250)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:921)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.roundRobinAssignment(BaseLoadBalancer.java:860)
>   at 
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2482)
>   at 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:282)
>   at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> {noformat}
> After that, region is left in limbo and is never reassigned.
> {noformat}
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.HMaster: Client=hrt_qa//68.142.246.29 move 
> hri=IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.,
>  src=hor13n19.gq1.ygridcore.net,60020,1393341563552, 
> dest=hor13n13.gq1.ygridcore.net,60020,139334275, running balancer
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.AssignmentManager: Ignored moving region not assigned: {ENCODED => 
> 24d68aa7239824e42390a77b7212fcbf, NAME => 
> 'IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.',
>  STARTKEY => '\x80\x06\x1A', ENDKEY => ''}, {24d68aa7239824e42390a77b7212fcbf 
> state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:35:26,586 DEBUG 
> [hor13n12.gq1.ygridcore.net,6,1393341917402-BalancerChore] 
> master.HMaster: Not running balancer because 1 region(s) in transition: 
> {24d68aa7239824e42390a77b7212fcbf={24d68aa7239824e42390a77b7212fcbf

[jira] [Updated] (HBASE-10632) Region lost in limbo after ArrayIndexOutOfBoundsException during assignment

2014-02-28 Thread Enis Soztutar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-10632:
--

Fix Version/s: 0.98.1
   0.96.2

> Region lost in limbo after ArrayIndexOutOfBoundsException during assignment
> ---
>
> Key: HBASE-10632
> URL: https://issues.apache.org/jira/browse/HBASE-10632
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: hbase-10070
>Reporter: Nick Dimiduk
>Assignee: Enis Soztutar
> Fix For: 0.96.2, 0.98.1, 0.99.0, hbase-10070
>
> Attachments: hbase-10632_v1.patch
>
>
> Discovered while running IntegrationTestBigLinkedList. Region 
> 24d68aa7239824e42390a77b7212fcbf is scheduled for move from hor13n19 to 
> hor13n13. During the process an exception is thrown.
> {noformat}
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioning {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} will be handled by SSH 
> for hor13n19.gq1.ygridcore.net,60020,1393341563552
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning 7 region(s) that 
> hor13n19.gq1.ygridcore.net,60020,1393341563552 was carrying (and 0 regions(s) 
> that were opening on this server)
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning region with rs = 
> {24d68aa7239824e42390a77b7212fcbf state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} and deleting zk node 
> if exists
> 2014-02-25 15:30:42,623 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioned {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} to 
> {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> 2014-02-25 15:30:42,623 DEBUG [AM.ZK.Worker-pool2-t46] 
> master.AssignmentManager: Znode 
> IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.
>  deleted, state: {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, 
> ts=1393342242623, server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:30:43,993 ERROR [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> executor.EventHandler: Caught throwable while processing event 
> M_SERVER_SHUTDOWN
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:250)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:921)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.roundRobinAssignment(BaseLoadBalancer.java:860)
>   at 
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2482)
>   at 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:282)
>   at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> {noformat}
> After that, region is left in limbo and is never reassigned.
> {noformat}
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.HMaster: Client=hrt_qa//68.142.246.29 move 
> hri=IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.,
>  src=hor13n19.gq1.ygridcore.net,60020,1393341563552, 
> dest=hor13n13.gq1.ygridcore.net,60020,139334275, running balancer
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.AssignmentManager: Ignored moving region not assigned: {ENCODED => 
> 24d68aa7239824e42390a77b7212fcbf, NAME => 
> 'IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.',
>  STARTKEY => '\x80\x06\x1A', ENDKEY => ''}, {24d68aa7239824e42390a77b7212fcbf 
> state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:35:26,586 DEBUG 
> [hor13n12.gq1.ygridcore.net,6,1393341917402-BalancerChore] 
> master.HMaster: Not running balancer because 1 region(s) in transition: 
> {24d68aa7239824e42390a77b7212fcbf={24d68aa7239824e42390a77b7212fcbf 
> state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}}
> ...
> 2014-02-25 15:35:51,945 DEBUG [FifoRpc

[jira] [Updated] (HBASE-10632) Region lost in limbo after ArrayIndexOutOfBoundsException during assignment

2014-02-27 Thread Enis Soztutar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-10632:
--

Status: Patch Available  (was: Open)

> Region lost in limbo after ArrayIndexOutOfBoundsException during assignment
> ---
>
> Key: HBASE-10632
> URL: https://issues.apache.org/jira/browse/HBASE-10632
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: hbase-10070
>Reporter: Nick Dimiduk
>Assignee: Enis Soztutar
> Fix For: 0.99.0, hbase-10070
>
> Attachments: hbase-10632_v1.patch
>
>
> Discovered while running IntegrationTestBigLinkedList. Region 
> 24d68aa7239824e42390a77b7212fcbf is scheduled for move from hor13n19 to 
> hor13n13. During the process an exception is thrown.
> {noformat}
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioning {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} will be handled by SSH 
> for hor13n19.gq1.ygridcore.net,60020,1393341563552
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning 7 region(s) that 
> hor13n19.gq1.ygridcore.net,60020,1393341563552 was carrying (and 0 regions(s) 
> that were opening on this server)
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning region with rs = 
> {24d68aa7239824e42390a77b7212fcbf state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} and deleting zk node 
> if exists
> 2014-02-25 15:30:42,623 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioned {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} to 
> {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> 2014-02-25 15:30:42,623 DEBUG [AM.ZK.Worker-pool2-t46] 
> master.AssignmentManager: Znode 
> IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.
>  deleted, state: {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, 
> ts=1393342242623, server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:30:43,993 ERROR [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> executor.EventHandler: Caught throwable while processing event 
> M_SERVER_SHUTDOWN
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:250)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:921)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.roundRobinAssignment(BaseLoadBalancer.java:860)
>   at 
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2482)
>   at 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:282)
>   at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> {noformat}
> After that, region is left in limbo and is never reassigned.
> {noformat}
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.HMaster: Client=hrt_qa//68.142.246.29 move 
> hri=IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.,
>  src=hor13n19.gq1.ygridcore.net,60020,1393341563552, 
> dest=hor13n13.gq1.ygridcore.net,60020,139334275, running balancer
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.AssignmentManager: Ignored moving region not assigned: {ENCODED => 
> 24d68aa7239824e42390a77b7212fcbf, NAME => 
> 'IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.',
>  STARTKEY => '\x80\x06\x1A', ENDKEY => ''}, {24d68aa7239824e42390a77b7212fcbf 
> state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:35:26,586 DEBUG 
> [hor13n12.gq1.ygridcore.net,6,1393341917402-BalancerChore] 
> master.HMaster: Not running balancer because 1 region(s) in transition: 
> {24d68aa7239824e42390a77b7212fcbf={24d68aa7239824e42390a77b7212fcbf 
> state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}}
> ...
> 2014-02-25 15:35:51,945 DEBUG [FifoRpcScheduler.handler1-thread-1

[jira] [Updated] (HBASE-10632) Region lost in limbo after ArrayIndexOutOfBoundsException during assignment

2014-02-27 Thread Enis Soztutar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-10632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-10632:
--

Attachment: hbase-10632_v1.patch

Nice find Nick! 

The patch fixes three issues, only the first is the root cause of the stack 
trace above: 
 - A subtle issue when multiple serverNames are passed with same host and port, 
their regions are combined. The length calculation was wrong. 
 - region locations coming from RegionLocationFinder are not used correctly in 
SLB, since the server indexes cannot be found because of we should have been 
using ServerName.getHostAndPort() 
 - LocalityBasedPicker was returning the wrong server index for the best 
location with highest block locality. 

> Region lost in limbo after ArrayIndexOutOfBoundsException during assignment
> ---
>
> Key: HBASE-10632
> URL: https://issues.apache.org/jira/browse/HBASE-10632
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: hbase-10070
>Reporter: Nick Dimiduk
>Assignee: Enis Soztutar
> Fix For: 0.99.0, hbase-10070
>
> Attachments: hbase-10632_v1.patch
>
>
> Discovered while running IntegrationTestBigLinkedList. Region 
> 24d68aa7239824e42390a77b7212fcbf is scheduled for move from hor13n19 to 
> hor13n13. During the process an exception is thrown.
> {noformat}
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioning {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} will be handled by SSH 
> for hor13n19.gq1.ygridcore.net,60020,1393341563552
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning 7 region(s) that 
> hor13n19.gq1.ygridcore.net,60020,1393341563552 was carrying (and 0 regions(s) 
> that were opening on this server)
> 2014-02-25 15:30:42,613 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> handler.ServerShutdownHandler: Reassigning region with rs = 
> {24d68aa7239824e42390a77b7212fcbf state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} and deleting zk node 
> if exists
> 2014-02-25 15:30:42,623 INFO  [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> master.RegionStates: Transitioned {24d68aa7239824e42390a77b7212fcbf 
> state=OPENING, ts=1393342207107, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552} to 
> {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, ts=1393342242623, 
> server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> 2014-02-25 15:30:42,623 DEBUG [AM.ZK.Worker-pool2-t46] 
> master.AssignmentManager: Znode 
> IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.
>  deleted, state: {24d68aa7239824e42390a77b7212fcbf state=OFFLINE, 
> ts=1393342242623, server=hor13n19.gq1.ygridcore.net,60020,1393341563552}
> ...
> 2014-02-25 15:30:43,993 ERROR [MASTER_SERVER_OPERATIONS-hor13n12:6-4] 
> executor.EventHandler: Caught throwable while processing event 
> M_SERVER_SHUTDOWN
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:250)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:921)
>   at 
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.roundRobinAssignment(BaseLoadBalancer.java:860)
>   at 
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2482)
>   at 
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:282)
>   at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> {noformat}
> After that, region is left in limbo and is never reassigned.
> {noformat}
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.HMaster: Client=hrt_qa//68.142.246.29 move 
> hri=IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.,
>  src=hor13n19.gq1.ygridcore.net,60020,1393341563552, 
> dest=hor13n13.gq1.ygridcore.net,60020,139334275, running balancer
> 2014-02-25 15:35:11,581 INFO  [FifoRpcScheduler.handler1-thread-6] 
> master.AssignmentManager: Ignored moving region not assigned: {ENCODED => 
> 24d68aa7239824e42390a77b7212fcbf, NAME => 
> 'IntegrationTestBigLinkedList,\x80\x06\x1A,1393342105093.24d68aa7239824e42390a77b7212fcbf.',
>  STARTKEY => '\x80\x06\x1A', ENDKE