Since ours is production cluster we cant restart master. In our test cluster I tested this scenario, and it got resolved after restarting master. Other than restarting master I couldn't find any solution. Thanks,Sandeep.
> From: [email protected] > Date: Wed, 4 Mar 2015 14:55:03 +0100 > Subject: Re: Where is HBase failed servers list stored > To: [email protected] > > If I understand the issue correctly, restarting the master should solve the > problem. > > On Wed, Mar 4, 2015 at 5:55 AM, Ted Yu <[email protected]> wrote: > > > Please see HBASE-13067 Fix caching of stubs to allow IP address changes of > > restarted remote servers > > > > Cheers > > > > On Tue, Mar 3, 2015 at 8:26 PM, Sandeep L <[email protected]> > > wrote: > > > > > Hi nkeywal, > > > While trying to get more details about this issue I got to know that > > > HMaster is trying to connect to wrong IP Address. > > > Here is exact issue: > > > Due to some unavoidable reason we are forced to change IP Address of > > > regionsserver & then updated new IP Address in /etc/hosts file across all > > > HBase servers. I started RegionServer from master with start-hbase.sh > > > scripts & jps output in regionserver shows it's(regionserver process) up > > > and running. > > > But when running hbase balancer HMaster is trying to connect to old IP > > > Address instead of new IP Address. > > > One more thing here is when I checked regionserver status on 60010 port > > > its showing as up and running. > > > Thanks,Sandeep. > > > > > > > From: [email protected] > > > > Date: Tue, 3 Mar 2015 19:01:01 +0100 > > > > Subject: Re: Where is HBase failed servers list stored > > > > To: [email protected] > > > > > > > > It's in local memory. When HBase cannot connect to a server, it puts it > > > > into the "failedServerList" for 2 seconds. This is to avoid having all > > > the > > > > threads going into a potentially long socket timeout. Are you sure that > > > you > > > > can connect from the master to this machine/port? > > > > > > > > You can change the time it stays in the list with > > > > hbase.ipc.client.failed.servers.expiry (in milliseconds), but it should > > > not > > > > help. > > > > > > > > You should have another exception before this one in the logs (the one > > > that > > > > initially put this region server in this failedServerList). > > > > > > > > On Tue, Mar 3, 2015 at 12:08 PM, Sandeep L <[email protected]> > > > > wrote: > > > > > > > > > Hi, > > > > > While trying to run hbase balancer I am getting error message as > > "This > > > > > server is in the failed servers list".Due to this cluster is not > > > getting > > > > > balanced. > > > > > Even though regionserver is up and running hmaster is unable to > > > connect to > > > > > it. > > > > > The odd thing here is hmaster is able to start regionserver and it is > > > > > detected as up and running but unable to assign regions. > > > > > Can some one suggest any solution for this. > > > > > Following is full stack > > > > > trace:org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: > > This > > > > > server is in the failed servers list: host1/192.168.2.20:60020 at > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) > > > > > at > > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) > > > > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) > > > at > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:999) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447) > > > > > at > > > > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260) > > > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > Thanks,Sandeep. > > > > > > > >
