Hi I'm using hbase0.96.0.
I found client can't put data suddenly and hmaster hungs. Then I shutdown the hmaster and start a new hmaster, then the client back to normal. I found this logs in the new hmaster . It seem like meta server hungs and hmaster stop the meta server. 2014-04-29 15:32:21,530 INFO [master:hadoop1:60000] catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=hadoop77,60020,1396606457005, exception=java.net.SocketTimeoutException: Call to hadoop77/192.168.1.87:60020 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.1.123:33117 remote=hadoop77/192.168.1.87:60020] 2014-04-29 15:32:21,532 INFO [master:hadoop1:60000] master.HMaster: Forcing expire of hadoop77,60020,1396606457005 I can't find why meta server hungs .I found this in meta server log 2014-04-29 13:53:55,637 INFO [regionserver60020.leaseChecker] regionserver.HRegionServer: Scanner 8206938292079629452 lease expired on region hbase:meta,,1.1588230740 2014-04-29 13:53:56,632 INFO [regionserver60020.leaseChecker] regionserver.HRegionServer: Scanner 1111451530521284267 lease expired on region hbase:meta,,1.1588230740 2014-04-29 13:53:56,733 INFO [regionserver60020.leaseChecker] regionserver.HRegionServer: Scanner 516152687416913803 lease expired on region hbase:meta,,1.1588230740 2014-04-29 13:53:56,733 INFO [regionserver60020.leaseChecker] regionserver.HRegionServer: Scanner -2651411216936596082 lease expired on region hbase:meta,,1.1588230740 any suggestion will be appreciated. Thanks.
