I have a application which query/inserts data in to Accumulo
continuously. But the ingestion performance degrades after a day and
continuously degrades by a considerable factor. I scanned all the hadoop
and Accumulo logs but only found a warning in my two tservers logs after
some inserts regularly.
[hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
(threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS
reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0,
targets:
[DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],
DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],
DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]
After monitoring the query time, It seems to be increased(for ex from 30
ms to 3 sec)after a day and continues. I also get the connection reset
exception regularly in Tserver logs in both read and write operations.
Which I read that it comes when a connection is closed before scan or
write completes. But how does it affect?
015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer]
WARN : Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
at
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
at
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)
Please provide your inputs.
-Mohit Kaushik
