I shutdown hbase. Added some new nodes to hdfs, rebalanced. Also added those
nodes to hbase regionservers.
Then started hbase.
I am having this strange problem where the new nodes let's say host1 thru host4
gets repeatedly reported/added to the regionservers list.
Initially when I did a "report 'simple'" from the shell, it showed me 10 unique
hosts. Then within a matter of minutes it grew to 17 ( with the newly added
hosts repeating multiple times).
Also, the web UI failed with the following error.
##############
HTTP ERROR: 500
Trying to contact region server 192.168.130.63:60020 for region .META.,,1, row
'', but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2266)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1845)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
###############
Any insight into why the regions get repeated multiple times. I did a hadoop
fsck / and it reports that all the blocks have been replicated 3 times ( the
configured value ).
Thanks
Arun
