One possible cause of this is a communication error with a remote tablet server. For example if the thrift call at TabletServerBatchReaderIterator.java line 665 [1] failed with a comm error, then it would try to close the connection at line 686. Then when the close fails (because its a busted socket), it seems that thrift logs a warn[2]. I think the batch scanner will retry in this case. I suspect line 665 would throw a TTransportException which would cause a retry. Do you know if the batch scanner continued to operate? Did any exceptions propagate to code using the batch scanner?
[1]: https://github.com/apache/accumulo/blob/rel/1.7.2/core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java#L665 [2]: https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L112 On Wed, Apr 19, 2017 at 12:38 PM, David Boyd <db...@incadencecorp.com> wrote: > All: > > I am getting this stack trace periodically based on no pattern I can > determine from my application. > > Is this a message I should be worried about? > > What is a technique to trace this back to my code and the cause? > > Obviously something is closing things before the thread closes it. > > >> 2017-04-19 12:33:53,423 | WARN | [batch scanner 19824- 8 looking up 1 >> ranges at accumulodev:9997] | (TIOStreamTransport.java:112) - Error closing >> output stream. >> java.io.IOException: The stream is closed >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) >> at >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> at java.io.FilterOutputStream.close(FilterOutputStream.java:158) >> at >> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) >> at >> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89) >> at >> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309) >> at >> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571) >> at >> org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151) >> at >> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686) >> at >> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349) >> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at >> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) >> at java.lang.Thread.run(Thread.java:745) > > > > -- > ========= mailto:db...@incadencecorp.com ============ > David W. Boyd > VP, Data Solutions > 10432 Balls Ford, Suite 240 > Manassas, VA 20109 > office: +1-703-552-2862 > cell: +1-703-402-7908 > ============== http://www.incadencecorp.com/ ============ > ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture > Chair ANSI/INCITS TC Big Data > Co-chair NIST Big Data Public Working Group Reference Architecture > First Robotic Mentor - FRC, FTC - www.iliterobotics.org > Board Member- USSTEM Foundation - www.usstem.org > > The information contained in this message may be privileged > and/or confidential and protected from disclosure. > If the reader of this message is not the intended recipient > or an employee or agent responsible for delivering this message > to the intended recipient, you are hereby notified that any > dissemination, distribution or copying of this communication > is strictly prohibited. If you have received this communication > in error, please notify the sender immediately by replying to > this message and deleting the material from any computer. > >
