Broken pipe when indexing in solr using ConcurrentUpdateSolrClient
Hello, Sometimes we have "broken pipe" error when indexing 40 000 000 of documents using ConcurrentUpdateSolrClient in java. I have find on google that's probably is because à time out. But in ConcurrentUpdateSolrClient we can't configure timeout. For example the last broken pie was for indexing 40 000 000 of documents and several broken pipe occurred and 16 000 files would'nt indexing. Thank you for helping. Yassin
Re: Broken pipe error
On 11/29/2016 4:47 AM, Srinivas Kashyap wrote: > Can somebody guide me how to resolve this issue? > > Some of the parameters for Tomcat set are : > > maxWait="15000" maxActive="1000" maxIdle="50". A broken pipe error usually means that the TCP connection was broken, but you didn't include enough of the stacktrace for me to say for sure. Normally this would mean that the client disconnected before the request completed, and that's usually due to a timeout. The most likely timeout to apply is a TCP socket timeout. Since you're not running in the Jetty that was included with Solr, I can't say for sure that it's not a timeout issue on the server side, because I have no way of knowing how the server is configured. What version of Solr? Running in Tomcat became an unsupported config (though still possible) with the release of 5.0, which was over a year ago. Thanks, Shawn
Broken pipe error
Hello, After starting the solr application and running full imports, running into this below error after a while: null:org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at org.apache.solr.util.FastWriter.flush(FastWriter.java:137) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:710) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:430) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina Can somebody guide me how to resolve this issue? Some of the parameters for Tomcat set are : maxWait="15000" maxActive="1000" maxIdle="50". Thanks and Regards, Srinivas DISCLAIMER: E-mails and attachments from TradeStone Software, Inc. are confidential. If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
Re: Java.net.socketexception: broken pipe Solr 4.10.2
I haven't had time to really take a look at this. But read a couple of articles regarding the hard commit and it actually makes sense. We were seeing tlogs in the multiple GBs during ingest. I will have some time in a couple of weeks to come back to testing indexing. Thanks for the help. Vy -- View this message in context: http://lucene.472066.n3.nabble.com/Java-net-socketexception-broken-pipe-Solr-4-10-2-tp4199484p4200498.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Java.net.socketexception: broken pipe Solr 4.10.2
On 4/13/2015 10:11 PM, vsilgalis wrote: just a couple of notes: this a 2 shard setup with 2 nodes per shard. Currently these are on VMs with 8 cores and 8GB of ram each (java max heap is ~5588mb but we usually never even get that high) backed by a NFS file store which we store the indexes on (netapp SAN with nfs exports on SAS disk). Broken pipe errors usually indicate that the client gave up waiting for the server and disconnected the TCP connection before the server completed processing and sent a response. This is frequently because of configured timeouts on the client. If reasonable timeouts are being exceeded, it's usually a performance problem. You haven't indicated how much disk space is occupied by the index data on each of these servers. There are also several other things that would be helpful to know. Please read this wiki page, then come back with any questions you might have, and I may also ask a question or two: http://wiki.apache.org/solr/SolrPerformanceProblems My immediate suspects are an OS disk cache that is too small, and/or problems with garbage collection pauses. These are two of the issues discussed on that wiki page. Thanks, Shawn
Re: Java.net.socketexception: broken pipe Solr 4.10.2
Right now index size is about 10GB on each shard (yes I could use more RAM), but I'm looking more for a step up then step down approach. I will try adding more RAM to these machines as my next step. 1. Zookeeper is external to these boxes in a three node cluster with more than enough RAM to keep everything off disk. 2. os disk cache, when I add more RAM I will just add it as RAM for the machine and not to the Java Heap unless that is something you recommend. 3. java heap looks good so far, GC is minimal as far as i can tell but I can look into this some more. 4. we do have 2 cores per machine, but the second core is a joke (10MB) note: zkClientTimeout is set to 30 for safety's sake. java settings: -XX:+CMSClassUnloadingEnabled-XX:+AggressiveOpts-XX:+ParallelRefProcEnabled-XX:+CMSParallelRemarkEnabled-XX:CMSMaxAbortablePrecleanTime=6000-XX:CMSTriggerPermRatio=80-XX:CMSInitiatingOccupancyFraction=50-XX:+UseCMSInitiatingOccupancyOnly-XX:CMSFullGCsBeforeCompaction=1-XX:PretenureSizeThreshold=64m-XX:+CMSScavengeBeforeRemark-XX:ParallelGCThreads=4-XX:ConcGCThreads=4-XX:+UseConcMarkSweepGC-XX:+UseParNewGC-XX:MaxTenuringThreshold=8-XX:TargetSurvivorRatio=90-XX:SurvivorRatio=4-XX:NewRatio=3-XX:-UseSuperWord-Xmx5588m-Xms1596m -- View this message in context: http://lucene.472066.n3.nabble.com/Java-net-socketexception-broken-pipe-Solr-4-10-2-tp4199484p4199561.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Java.net.socketexception: broken pipe Solr 4.10.2
We ran into this during our indexing process running on 4.10.3. After increasing zookeeper timeouts, client timeouts, socket timeouts, implementing retry logic on our loading process the thing that worked was to change the Hard Commit timing. We were performing a Hard Commit every 5 minutes and after a couple hours of loading data some of the shards would start going down because they would timeout with zookeeper and/or close connections. Changing the timeouts just moved the problem later in the ingest process. Through a combination of decreasing the hard commit timing to 15 seconds, and migrating to G1 garbage collect, we are able to prevent ingest failures. For us the periodic stop the world garbage collects were causing connections to be closed and other nasty things such as zookeeper timeouts that would cause recovery to kick in. (Soft commits are turned off until the full ingest/baseline completes). I believe until a Hard Commit is issued Solr keeps the data in memory which explains why we were experiencing nasty garbage collects. The other change we made which may have helped is that we ensured the socket timeouts were in sync between the jetty instance running Solr and the SolrJ loading the data. During some of our batch updates Solr would take a couple minutes to respond back which I believe in some instances the socket server side would be closed (maxIdleTime setting in Jetty). Hope this helps, Jaime Spicciati Thanks Jaime On Tue, Apr 14, 2015 at 9:26 AM, vsilgalis vsilga...@gmail.com wrote: Right now index size is about 10GB on each shard (yes I could use more RAM), but I'm looking more for a step up then step down approach. I will try adding more RAM to these machines as my next step. 1. Zookeeper is external to these boxes in a three node cluster with more than enough RAM to keep everything off disk. 2. os disk cache, when I add more RAM I will just add it as RAM for the machine and not to the Java Heap unless that is something you recommend. 3. java heap looks good so far, GC is minimal as far as i can tell but I can look into this some more. 4. we do have 2 cores per machine, but the second core is a joke (10MB) note: zkClientTimeout is set to 30 for safety's sake. java settings: -XX:+CMSClassUnloadingEnabled-XX:+AggressiveOpts-XX:+ParallelRefProcEnabled-XX:+CMSParallelRemarkEnabled-XX:CMSMaxAbortablePrecleanTime=6000-XX:CMSTriggerPermRatio=80-XX:CMSInitiatingOccupancyFraction=50-XX:+UseCMSInitiatingOccupancyOnly-XX:CMSFullGCsBeforeCompaction=1-XX:PretenureSizeThreshold=64m-XX:+CMSScavengeBeforeRemark-XX:ParallelGCThreads=4-XX:ConcGCThreads=4-XX:+UseConcMarkSweepGC-XX:+UseParNewGC-XX:MaxTenuringThreshold=8-XX:TargetSurvivorRatio=90-XX:SurvivorRatio=4-XX:NewRatio=3-XX:-UseSuperWord-Xmx5588m-Xms1596m -- View this message in context: http://lucene.472066.n3.nabble.com/Java-net-socketexception-broken-pipe-Solr-4-10-2-tp4199484p4199561.html Sent from the Solr - User mailing list archive at Nabble.com.
Java.net.socketexception: broken pipe Solr 4.10.2
We upgraded recently to Solr 4.10.2 from 4.2.1 and have been seeing errors regarding the dreaded broken pipe when doing our reindexing of all our content. Specifically: ERROR - 2015-04-13 17:09:12.310; org.apache.solr.update.StreamingSolrServers$1; error java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181) at org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:124) at org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:181) at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214) at org.apache.solr.common.util.FastOutputStream.write(FastOutputStream.java:84) at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:586) at org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:661) at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:240) at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153) at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:424) at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:274) at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153) at org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:509) at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:294) at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153) at org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:450) at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:282) at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153) at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:148) at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:242) at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153) at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:96) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:83) at org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:108) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner$1.writeTo(ConcurrentUpdateSolrServer.java:186) at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:69) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265) at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) which in turn leads to: WARN - 2015-04-13 17:09:53.586; org.apache.solr.update.processor.DistributedUpdateProcessor; Error sending update java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java
Re: Java.net.socketexception: broken pipe Solr 4.10.2
just a couple of notes: this a 2 shard setup with 2 nodes per shard. Currently these are on VMs with 8 cores and 8GB of ram each (java max heap is ~5588mb but we usually never even get that high) backed by a NFS file store which we store the indexes on (netapp SAN with nfs exports on SAS disk). -- View this message in context: http://lucene.472066.n3.nabble.com/Java-net-socketexception-broken-pipe-Solr-4-10-2-tp4199484p4199485.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr cloud shard goes down when after many broken pipe exceptions
First ClientAbortException comes, which is expected as there is timeout on client side with stack trace as follows Jun 30, 2013 2:24:30 PM org.apache.solr.common.SolrException log SEVERE: null:ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:369) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339) at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392) at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381) at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190) at org.apache.solr.util.FastWriter.flush(FastWriter.java:141) at org.apache.solr.util.FastWriter.write(FastWriter.java:55) at org.apache.solr.response.JSONWriter.writeStr(JSONResponseWriter.java:449) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:124) at org.apache.solr.response.JSONWriter.writeSolrDocument(JSONResponseWriter.java:355) at org.apache.solr.response.TextResponseWriter.writeSolrDocumentList(TextResponseWriter.java:222) at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:184) at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183) at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299) at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:95) at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:60) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:404) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:282) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:756) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:448) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:363) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:780) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:593) at org.apache.coyote.Response.doWrite(Response.java:560) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:364) ... 33 more The above exception comes a number of times followed by a shard getting down Jun 30, 2013 2:24:33 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: no servers hosting shard: at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:135) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303
Broken pipe
Any idea why I got a Broken pipe?
INFO - 2013-05-23 13:37:19.881; org.apache.solr.core.SolrCore;
[messages_shard3_replica1] webapp=/solr path=/select/
params={sort=score+descfl=id,smsc_module,smsc_modulekey,smsc_userid,smsc_ssid,smsc_description,smsc_description_ngram,smsc_content,smsc_content_ngram,smsc_courseid,smsc_lastdate,score,metadata_stream_size,metadata_stream_source_info,metadata_stream_name,metadata_stream_content_type,last_modified,author,title,subjectdebugQuery=truedefaultOperator=ANDindent=onstart=0q=(smsc_content:banaan+||+smsc_content_ngram:banaan+||+smsc_description:banaan+||+smsc_description_ngram:banaan)+%26%26+(smsc_lastdate:[2000-04-23T15:14:40Z+TO+2013-05-23T15:14:40Z])+%26%26+(smsc_ssid:9)collection=messageswt=xmlrows=50version=2.2}
hits=119 status=0 QTime=81108
ERROR - 2013-05-23 13:37:19.892; org.apache.solr.common.SolrException;
null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:155)
at
org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:85)
at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:41)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:117)
at
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:505)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
... 30 more
ERROR - 2013-05-23 13:37:19.893; org.apache.solr.common.SolrException;
null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419
AW: Broken pipe
This usually happens when the client sending the request to Solr has given up
waiting for the response (terminated the connection).
In your example, we see that the Solr query time is 81 seconds. Probably the
client issuing the request has a time-out of maybe 30 or 60 seconds.
André
Von: Arkadi Colson [ark...@smartbit.be]
Gesendet: Donnerstag, 23. Mai 2013 15:40
An: solr-user@lucene.apache.org
Betreff: Broken pipe
Any idea why I got a Broken pipe?
INFO - 2013-05-23 13:37:19.881; org.apache.solr.core.SolrCore;
[messages_shard3_replica1] webapp=/solr path=/select/
params={sort=score+descfl=id,smsc_module,smsc_modulekey,smsc_userid,smsc_ssid,smsc_description,smsc_description_ngram,smsc_content,smsc_content_ngram,smsc_courseid,smsc_lastdate,score,metadata_stream_size,metadata_stream_source_info,metadata_stream_name,metadata_stream_content_type,last_modified,author,title,subjectdebugQuery=truedefaultOperator=ANDindent=onstart=0q=(smsc_content:banaan+||+smsc_content_ngram:banaan+||+smsc_description:banaan+||+smsc_description_ngram:banaan)+%26%26+(smsc_lastdate:[2000-04-23T15:14:40Z+TO+2013-05-23T15:14:40Z])+%26%26+(smsc_ssid:9)collection=messageswt=xmlrows=50version=2.2}
hits=119 status=0 QTime=81108
ERROR - 2013-05-23 13:37:19.892; org.apache.solr.common.SolrException;
null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:155)
at
org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:85)
at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:41)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:117)
at
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:505)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
... 30 more
ERROR - 2013
Re: Broken pipe
Also happens (same reason) if you are behind a smart load-balance and
it decides to time out and fail over.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Thu, May 23, 2013 at 9:59 AM, André Widhani andre.widh...@digicol.de wrote:
This usually happens when the client sending the request to Solr has given up
waiting for the response (terminated the connection).
In your example, we see that the Solr query time is 81 seconds. Probably the
client issuing the request has a time-out of maybe 30 or 60 seconds.
André
Von: Arkadi Colson [ark...@smartbit.be]
Gesendet: Donnerstag, 23. Mai 2013 15:40
An: solr-user@lucene.apache.org
Betreff: Broken pipe
Any idea why I got a Broken pipe?
INFO - 2013-05-23 13:37:19.881; org.apache.solr.core.SolrCore;
[messages_shard3_replica1] webapp=/solr path=/select/
params={sort=score+descfl=id,smsc_module,smsc_modulekey,smsc_userid,smsc_ssid,smsc_description,smsc_description_ngram,smsc_content,smsc_content_ngram,smsc_courseid,smsc_lastdate,score,metadata_stream_size,metadata_stream_source_info,metadata_stream_name,metadata_stream_content_type,last_modified,author,title,subjectdebugQuery=truedefaultOperator=ANDindent=onstart=0q=(smsc_content:banaan+||+smsc_content_ngram:banaan+||+smsc_description:banaan+||+smsc_description_ngram:banaan)+%26%26+(smsc_lastdate:[2000-04-23T15:14:40Z+TO+2013-05-23T15:14:40Z])+%26%26+(smsc_ssid:9)collection=messageswt=xmlrows=50version=2.2}
hits=119 status=0 QTime=81108
ERROR - 2013-05-23 13:37:19.892; org.apache.solr.common.SolrException;
null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:155)
at
org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:85)
at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:41)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480
Re: Broken pipe error
I had the same problem with jetty. It turned out that broken pipe happens when application disconnects from jetty. In my case I was using php client and it had 10 sec restriction in curl request. When solr takes more than 10 sec to respond, curl automatically disconnected from jetty. Hope this can help. Alex. -Original Message- From: Jason hialo...@gmail.com To: solr-user solr-user@lucene.apache.org Sent: Mon, Jul 2, 2012 7:41 pm Subject: Broken pipe error Hi, all We're independently running three search servers. One of three servers has bigger index size and more connection users than the others. Except that, all configurations are same. Problem is that server sometimes occurs broken pipe error. But I don't know what problem is. Please give some ideas. Thanks in advance. Jason error message below... === 2012-07-03 10:42:56,753 [http-8080-exec-3677] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:ClientAbortException: java.io.IOException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at org.apache.solr.util.FastWriter.flush(FastWriter.java:115) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:402) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:279) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2262) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:116) at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:93) at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:156) at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:460) at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:804) at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:644) at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:46) at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:829) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(InternalNioOutputBuffer.java:610) at org.apache.coyote.Response.doWrite(Response.java:560) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353) ... 25 more -- View this message in context: http://lucene.472066.n3.nabble.com/Broken-pipe-error
RE: Broken pipe error
I also had this problem on solr/tomcat and finally saw the errors were coming from my application side disconnecting from solr after a timeout. This was happening when solr was busy doing an optimize and thus not responding quickly enough. Initially when I saw this in the logs, I was quite worried until I realized the source of the problem. Robi -Original Message- From: alx...@aim.com [mailto:alx...@aim.com] Sent: Tuesday, July 03, 2012 10:38 AM To: solr-user@lucene.apache.org Subject: Re: Broken pipe error I had the same problem with jetty. It turned out that broken pipe happens when application disconnects from jetty. In my case I was using php client and it had 10 sec restriction in curl request. When solr takes more than 10 sec to respond, curl automatically disconnected from jetty. Hope this can help. Alex. -Original Message- From: Jason hialo...@gmail.com To: solr-user solr-user@lucene.apache.org Sent: Mon, Jul 2, 2012 7:41 pm Subject: Broken pipe error Hi, all We're independently running three search servers. One of three servers has bigger index size and more connection users than the others. Except that, all configurations are same. Problem is that server sometimes occurs broken pipe error. But I don't know what problem is. Please give some ideas. Thanks in advance. Jason error message below... === 2012-07-03 10:42:56,753 [http-8080-exec-3677] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:ClientAbortException: java.io.IOException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at org.apache.solr.util.FastWriter.flush(FastWriter.java:115) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:402) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:279) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2262) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69) at sun.nio.ch.IOUtil.write(IOUtil.java:40) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:116) at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:93) at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:156) at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:460) at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:804) at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:644) at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:46) at org.apache.coyote.http11.InternalNioOutputBuffer
Re: Solr TransformerException, SocketException: Broken pipe
Hi Guys, We are experiencing SEVERE exceptions in SOLR (stacktrace below) Please let me know if anyone has experienced this and have some insight / pointers on to where and what should I look for to resolve this. ERROR [solr.servlet.SolrDispatchFilter] - : java.io.IOException: XSLT transformation error After the exception the SOLR goes in Unstable state and the response time increases from less than 50ms to more than 5000ms. Thanks, Bhawna Hello, I have the same strange exception on my index. Did you solved it already or have any usefull advices? best regards JM
Solr TransformerException, SocketException: Broken pipe
Hi Guys, We are experiencing SEVERE exceptions in SOLR (stacktrace below) Please let me know if anyone has experienced this and have some insight / pointers on to where and what should I look for to resolve this. ERROR [solr.servlet.SolrDispatchFilter] - : java.io.IOException: XSLT transformation error After the exception the SOLR goes in Unstable state and the response time increases from less than 50ms to more than 5000ms. Thanks, Bhawna Stack Trace: ERROR [solr.servlet.SolrDispatchFilter] - [http-bio-8080-exec-10138] : java.io.IOException: XSLT transformation error at org.apache.solr.response.XSLTResponseWriter.write(XSLTResponseWriter.java:108) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:340) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:261) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:278) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: javax.xml.transform.TransformerException: ClientAbortException: java.net.SocketException: Broken pipe at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:719) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313) at org.apache.solr.response.XSLTResponseWriter.write(XSLTResponseWriter.java:106) ... 17 more Caused by: ClientAbortException: java.net.SocketException: Broken pipe at com.sun.org.apache.xml.internal.serializer.ToStream.flushWriter(ToStream.java:299) at com.sun.org.apache.xml.internal.serializer.ToXMLStream.endDocument(ToXMLStream.java:194) at com.sun.org.apache.xml.internal.serializer.ToUnknownStream.endDocument(ToUnknownStream.java:825) at dev.transform() at com.sun.org.apache.xalan.internal.xsltc.runtime.AbstractTranslet.transform(AbstractTranslet.java:603) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:709) ... 19 more Caused by: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:373) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:299) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:103) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:115) at com.sun.org.apache.xml.internal.serializer.ToStream.flushWriter(ToStream.java:294) ... 24 more Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:218) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:437) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:243) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119) at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:190) at org.apache.coyote.Response.doWrite(Response.java:533) at org.apache.catalina.connector.OutputBuffer.realWriteBytes
jetty error, broken pipe
Hello, I use solr 3.4 with jetty that is included in it. Periodically, I see this error in the jetty output SEVERE: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:296) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) ... ... ... Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 25 more 2011-11-19 20:50:00.060:WARN::Committed before 500 null||org.mortbay.jetty.EofException|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)|?at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)|?at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)|?at sun.nio.cs.StreamEncoder.implFlush(S I searched web and the only advice I get is to upgrade to jetty 6.1, but I think the version included in solr is 6.1.26. Any advise is appreciated. Thanks. Alex.
Re: jetty error, broken pipe
It's not Jetty. It is broken TCP pipe due to client-side. It happens when client closes TCP connection. And I even had this problem with recent Tomcat 6. Problem disappeared after I explicitly tuned keep-alive at Tomcat, and started using monitoring thread with HttpClient and SOLRJ... Fuad Efendi http://www.tokenizer.ca Sent from my iPad On 2011-11-19, at 9:14 PM, alx...@aim.com wrote: Hello, I use solr 3.4 with jetty that is included in it. Periodically, I see this error in the jetty output SEVERE: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:296) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) ... ... ... Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 25 more 2011-11-19 20:50:00.060:WARN::Committed before 500 null||org.mortbay.jetty.EofException|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)|?at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)|?at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)|?at sun.nio.cs.StreamEncoder.implFlush(S I searched web and the only advice I get is to upgrade to jetty 6.1, but I think the version included in solr is 6.1.26. Any advise is appreciated. Thanks. Alex.
Re: jetty error, broken pipe
I found out that curl timeout was set to 10 and for queries taking longer than 10 sec it was closing connection to jetty. I noticed that when number of docs found is large solr returns results for about 20 sec. This is too long. I set caching to off but it did not help. I think solr spends too much time to find total number of docs. Is there a way to turn off this count? Thanks. Alex. -Original Message- From: Fuad Efendi f...@efendi.ca To: solr-user solr-user@lucene.apache.org Cc: solr-user solr-user@lucene.apache.org Sent: Sat, Nov 19, 2011 7:24 pm Subject: Re: jetty error, broken pipe It's not Jetty. It is broken TCP pipe due to client-side. It happens when client closes TCP connection. And I even had this problem with recent Tomcat 6. Problem disappeared after I explicitly tuned keep-alive at Tomcat, and started using monitoring thread with HttpClient and SOLRJ... Fuad Efendi http://www.tokenizer.ca Sent from my iPad On 2011-11-19, at 9:14 PM, alx...@aim.com wrote: Hello, I use solr 3.4 with jetty that is included in it. Periodically, I see this error in the jetty output SEVERE: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:296) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) ... ... ... Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 25 more 2011-11-19 20:50:00.060:WARN::Committed before 500 null||org.mortbay.jetty.EofException|?at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)|?at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)|?at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)|?at sun.nio.cs.StreamEncoder.implFlush(S I searched web and the only advice I get is to upgrade to jetty 6.1, but I think the version included in solr is 6.1.26. Any advise is appreciated. Thanks. Alex.
