We observe that one core the CPU was nearly 100%. Here is also some log from the client:
2014-07-14 01:11:11[htable-pool20-t21776:224529968] - [INFO] #2242, table=monitor-data, attempt=10/1 failed 79 ops, last exception: java.net.SocketTimeoutException: Call to hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.210.87.44:17831 remote=hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420] on hz01-noah-cloudera32.hz01.xxxx.com,8420,1405055027387, tracking started Mon Jul 14 01:03:48 CST 2014, retrying after 2005 ms, replay 79 ops. 2014-07-14 01:11:11[htable-pool21-t22740:224529969] - [INFO] #2421, table=monitor-data, attempt=10/1 failed 112 ops, last exception: java.net.SocketTimeoutException: Call to hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.210.87.44:17831 remote=hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420] on hz01-noah-cloudera32.hz01.xxxx.com,8420,1405055027387, tracking started Mon Jul 14 01:03:35 CST 2014, retrying after 2007 ms, replay 112 ops. 2014-07-14 01:11:11[htable-pool19-t21540:224529974] - [INFO] #2241, table=monitor-data, attempt=10/1 failed 82 ops, last exception: java.net.SocketTimeoutException: Call to hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.210.87.44:17831 remote=hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420] on hz01-noah-cloudera32.hz01.xxxx.com,8420,1405055027387, tracking started Mon Jul 14 01:03:36 CST 2014, retrying after 2002 ms, replay 82 ops. 2014-07-14 01:11:11[htable-pool30-t23007:224529976] - [INFO] #5324, table=monitor-data, attempt=10/1 failed 2 ops, last exception: java.net.SocketTimeoutException: Call to hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420 failed because java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.210.87.44:17831 remote=hz01-noah-cloudera32.hz01.xxxx.com/10.211.135.19:8420] on hz01-noah-cloudera32.hz01.xxxx.com,8420,1405055027387, tracking started Mon Jul 14 01:03:09 CST 2014, retrying after 2015 ms, replay 2 ops. -- View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-region-servers-refuse-connection-tp4061278p4061294.html Sent from the HBase User mailing list archive at Nabble.com.
