>
> 2014-08-14 21:35:16,740 WARN org.apache.hadoop.hbase.util.Sleeper: We
> slept
> 14912ms instead of 3000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
I would check your gc logs for long gc pauses.
On Fri, May 29, 2015 at 11:38 AM, rahul malviya <malviyarahul2...@gmail.com>
wrote:
> Hi All,
>
> In our cluster region server logs are filled with response too slow
> message. This is causing jobs to slow down. How can I debug what is the
> reason for this slowness.
>
> We have enabled short circuit reads and region server has 27GB RAM.
>
> Here is a trace when regionserver starts.
>
> Thu Aug 14 20:23:51 GMT 2014 Starting regionserver on nodex
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 966365
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 32768
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 966365
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> 2014-08-14 20:23:53,341 WARN org.apache.hadoop.conf.Configuration:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2014-08-14 20:23:53,342 WARN org.apache.hadoop.conf.Configuration:
> mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
> 2014-08-14 20:23:53,884 WARN org.apache.hadoop.conf.Configuration:
> slave.host.name is deprecated. Instead, use
> mapreduce.tasktracker.host.name
> 2014-08-14 20:24:03,999 WARN org.apache.hadoop.conf.Configuration:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-08-14 20:26:47,605 ERROR
> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: Inconsistent
> configuration. Previous configuration for using table name in metrics:
> true, new configuration: false
> 2014-08-14 20:28:23,491 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":18725,"call":"next(-8041903839443097981, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.248:58716
>
> ","starttimems":1408048084720,"queuetimems":0,"class":"HRegionServer","responsesize":5031595,"method":"next"}
> 2014-08-14 21:35:16,740 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 14912ms instead of 3000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2014-08-14 21:42:28,477 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":16968,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:36657
>
> ","starttimems":1408052531504,"queuetimems":0,"class":"HRegionServer","responsesize":1959532,"method":"next"}
> 2014-08-14 21:42:56,923 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":10591,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:40818
>
> ","starttimems":1408052566327,"queuetimems":1,"class":"HRegionServer","responsesize":2987578,"method":"next"}
> 2014-08-14 21:44:24,372 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":10656,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:41993
>
> ","starttimems":1408052653710,"queuetimems":1,"class":"HRegionServer","responsesize":3039779,"method":"next"}
> 2014-08-14 21:45:50,598 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":12418,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:45197
>
> ","starttimems":1408052738174,"queuetimems":10,"class":"HRegionServer","responsesize":2476903,"method":"next"}
> 2014-08-14 21:46:15,187 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":23766,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:49425
>
> ","starttimems":1408052751414,"queuetimems":0,"class":"HRegionServer","responsesize":5681175,"method":"next"}
> 2014-08-14 21:47:09,041 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":12320,"call":"next(5487686201525374976, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.249:50269
>
> ","starttimems":1408052816698,"queuetimems":1,"class":"HRegionServer","responsesize":2986949,"method":"next"}
> 2014-08-14 21:49:23,833 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":11389,"call":"next(1227841280814011139, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.122:41976
>
> ","starttimems":1408052952415,"queuetimems":0,"class":"HRegionServer","responsesize":3160025,"method":"next"}
> 2014-08-14 21:49:23,869 WARN org.apache.hadoop.ipc.HBaseServer: Exception
> while changing ops : java.nio.channels.CancelledKeyException
> 2014-08-14 21:49:23,900 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":11428,"call":"next(9103372947568217267, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.41:35241
>
> ","starttimems":1408052952469,"queuetimems":0,"class":"HRegionServer","responsesize":1809158,"method":"next"}
> 2014-08-14 21:49:23,902 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":11415,"call":"next(-3120240140302998196, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.195:46046
>
> ","starttimems":1408052952468,"queuetimems":0,"class":"HRegionServer","responsesize":1826929,"method":"next"}
> 2014-08-14 21:49:24,050 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":11438,"call":"next(3799907609071248384, 10), rpc
> version=1, client version=29, methodsFingerPrint=-1368823753","client":"
> 17.170.176.154:42797
>
> ","starttimems":1408052952459,"queuetimems":0,"class":"HRegionServer","responsesize":2628568,"method":"next"}
> 2014-08-14 21:49:24,057 WARN org.apache.hadoop.ipc.HBaseServer:
> (responseTooSlow):
> {"processingtimems":11843,"call":"next(-1679362783893333095, 10), rpc
> version=1, client version=29,
> methodsFingerPrint=-1368823753","client":"17.170
>
> Thanks,
> Rahul
>
