On Wed, Jun 8, 2011 at 4:39 PM, Abhijit Pol <[email protected]> wrote:
> Recently we observed that our "get" latencies keep increasing over the
> period (and eventually flatten out at higher value) and if we restart hbase
> server, latencies go back to good state (low values) and start increasing
> again.
>
What happens if you flush that table/region when its slow (you can do
it from the shell). Does the latency go back down?
Are there lots of storefiles under these regions? (Check the fs. Do a
lsr on region you know is slow). If lots of storefiles, if you major
compact the region does the latency go back down?
If you look in your regionserver logs, what do the stats on your block
cache look like? Do the cache hits climb over time?
> 2011-06-08 19:xx:xx,188 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call get([B@16d0ea6f, row=adx:1CAESEInUuWF-DYppoEm8dJh5pFQ,
> maxVersions=1, cacheBlocks=true, timeRange=[0,9223372036854775807),
> families={(family=active_columns, columns=ALL}) from xx.xx.xx.xxx:34406:
> output error
> 2011-06-08 19:xx:xx,189 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 311 on 60020 caught: java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1336)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:724)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:789)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1080)
>
>
The client may have gone away because we took too long to process the
request. How many handlers are you running? Maybe the requests are
backing up in rpc queues?
Why 0.90.0 and not 0.90.3 (has some fixes).
> since server restart make things look good, is this might be related to
> minor compaction & block cache?
>
Give us some answers to a few of the above questions. Might help us
narrow in on whats going on.
St.Ack
