Re: Re: Hbase Dead region Server
Thanks Every one.It turned out that I there were a few empty wal directories corresponding to the dead region servers.Moved them out of /hbase and failed over the master.Things started working fine after that. -Nishanth On Mon, Nov 3, 2014 at 10:25 PM, yeweichen2...@gmail.com yeweichen2...@gmail.com wrote: Nishanth, What version of HBase you are using? You can try clear the ZNode about regionserver list in zookeeper /hbase/ and then restart HMaster. -- yeweichen2...@gmail.com *From:* Nishanth S nishanth.2...@gmail.com *Date:* 2014-11-04 02:32 *To:* user user@hbase.apache.org *Subject:* Re: Hbase Dead region Server Thanks Pere. I just did that and still has the dead region server showing up in Master UI as well as in status command.I have replication turned on in hbase and seeing few issues.Below is the stack trace I am seeing. 2014-11-03 18:31:00,215 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) 2014-11-03 18:31:00,459 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle p...@whisper.sh wrote: Nishanth, In my experience the only way I have been able to clear the dead region servers is to restart the master daemon. -Pere On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S nishanth.2...@gmail.com wrote: Hey folks, How do I remove a dead region server?.I manually failed over the hbase master but this is still appearing in master UI and also on the status command that I run. Thanks, Nishan
Re: Hbase Dead region Server
Nishanth, In my experience the only way I have been able to clear the dead region servers is to restart the master daemon. -Pere On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S nishanth.2...@gmail.com wrote: Hey folks, How do I remove a dead region server?.I manually failed over the hbase master but this is still appearing in master UI and also on the status command that I run. Thanks, Nishan
Re: Hbase Dead region Server
Thanks Pere. I just did that and still has the dead region server showing up in Master UI as well as in status command.I have replication turned on in hbase and seeing few issues.Below is the stack trace I am seeing. 2014-11-03 18:31:00,215 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) 2014-11-03 18:31:00,459 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle p...@whisper.sh wrote: Nishanth, In my experience the only way I have been able to clear the dead region servers is to restart the master daemon. -Pere On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S nishanth.2...@gmail.com wrote: Hey folks, How do I remove a dead region server?.I manually failed over the hbase master but this is still appearing in master UI and also on the status command that I run. Thanks, Nishan
Re: Hbase Dead region Server
Hi Pere and Nishanth, In master branch i developt a bash script to same problem. Its name is considerAsDead.sh [1] It mark as dead and start the recovery process. [1] https://github.com/apache/hbase/blob/master/bin/considerAsDead.sh Talat On Nov 3, 2014 8:32 PM, Nishanth S nishanth.2...@gmail.com wrote: Thanks Pere. I just did that and still has the dead region server showing up in Master UI as well as in status command.I have replication turned on in hbase and seeing few issues.Below is the stack trace I am seeing. 2014-11-03 18:31:00,215 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) 2014-11-03 18:31:00,459 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle p...@whisper.sh wrote: Nishanth, In my experience the only way I have been able to clear the dead region servers is to restart the master daemon. -Pere On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S nishanth.2...@gmail.com wrote: Hey folks, How do I remove a dead region server?.I manually failed over the hbase master but this is still appearing in master UI and also on the status command that I run. Thanks, Nishan
Re: Re: Hbase Dead region Server
Thanks Pere. I just did that and still has the dead region server showing up in Master UI as well as in status command.I have replication turned on in hbase and seeing few issues.Below is the stack trace I am seeing. 2014-11-03 18:31:00,215 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) 2014-11-03 18:31:00,459 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of a local or network error: java.io.IOException: No replication sinks are available at org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350) On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle p...@whisper.sh wrote: Nishanth, In my experience the only way I have been able to clear the dead region servers is to restart the master daemon. -Pere On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S nishanth.2...@gmail.com wrote: Hey folks, How do I remove a dead region server?.I manually failed over the hbase master but this is still appearing in master UI and also on the status command that I run. Thanks, Nishan
