Mohit, Were you able to figure this problem out? Do you have enough info to know if its a hdfs, Accumulo, network, or OS problem?
Keith On Mon, Nov 9, 2015 at 7:12 AM, mohit.kaushik <[email protected]> wrote: > > I have a application which query/inserts data in to Accumulo continuously. > But the ingestion performance degrades after a day and continuously > degrades by a considerable factor. I scanned all the hadoop and Accumulo > logs but only found a warning in my two tservers logs after some inserts > regularly. > > [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms > (threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS > reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0, > targets: > [DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK], > DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK], > DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]] > > > After monitoring the query time, It seems to be increased(for ex from 30 > ms to 3 sec)after a day and continues. I also get the connection reset > exception regularly in Tserver logs in both read and write operations. > Which I read that it comes when a connection is closed before scan or write > completes. But how does it affect? > > 015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer] > WARN : Got an IOException in internalRead! > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > at > org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141) > at > org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537) > at > org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338) > at > org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203) > at > org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227) > at > org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183) > > > Please provide your inputs. > > -Mohit Kaushik > > >
