Re: java.net.SocketException: Connection reset
I don't see anything out of the ordinary thus far, except your heap looks a little big. I usually run with 6-7gb. I'm wondering if maybe you're running into a juliet pause and that's causing your sockets to time out. Have you gathered any GC stats? Also, what are you doing with respect to commits and optimizes? Michael Della Bitta Applications Developer o: +1 646 532 3062 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY 10017 t: @appinions https://twitter.com/Appinions | g+: plus.google.com/appinions https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts w: appinions.com http://www.appinions.com/ On Fri, Jul 4, 2014 at 5:22 PM, heaven aheave...@gmail.com wrote: Today this had happened again + this one: null:java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181) at org.apache.http.impl.io.ChunkedOutputStream.flushCache(ChunkedOutputStream.java:111) at org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:193) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner$1.writeTo(ConcurrentUpdateSolrServer.java:206) at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:69) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265) at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Previously we had all 4 instances on a single node so I thought these errors might be result of high load. Like if some request taking too long to complete or something like that. And we always had missing docs in the index or vise verse some docs remains in the index when they shouldn't (even though it is supposed to recover from the log and our index queue never remove docs from it until it gets a successful response from Solr). But now we run shards and replicas on separate nodes with lots of resources and a very fast disk storage. And it still causes weird errors. It seems Solr is buggy as hell, that's my impression after a few years of usage. And it doesn't get better in this aspect, these errors follow us from the very beginning. -- View this message in context: http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519p4145675.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: java.net.SocketException: Connection reset
Yeah. the heap is huge, need to optimize the caches. It was 8Gb previously,
had to increase because there were out of memory errors. Using
ConcMarkSweepGC, which is supposed to not lock the world.
Had to disable optimize (previously we did so by a cron task) because the
index is big and optimize has bad impact on performance and resources usage.
We're using auto and soft commits only:
autoCommit
maxDocs25000/maxDocs
maxTime30/maxTime
openSearcherfalse/openSearcher
/autoCommit
autoSoftCommit
maxTime1/maxTime
/autoSoftCommit
I was thinking we may reach some system limits, but netstat doesn't show
anything suspicious:
alex@solr1:~$ netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
137 CLOSE_WAIT
24 ESTABLISHED
9 LISTEN
77 TIME_WAIT
It is also not clear where did those errors happen. It would be useful for
users (I mean for those not familiar with Solr development) if instead of
putting the entire backtrace (or in addition to it) Solr logged user
readable messages. Like: Error while sending response to the client
#{client_ip:client_port}, or when sending updates to replica
#{replica_ip:replica_port}. Because now those errors are pretty confusing.
Best,
Alex
--
View this message in context:
http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519p4145894.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: java.net.SocketException: Connection reset
Even i am facing same issue. AFTER doing a server restart again indexing can run fine once, but for second time same issue. On 3 Jul 2014 23:37, heaven aheave...@gmail.com wrote: Hi, trying DigitalOcean for Solr, everything seems well, except sometimes I see these errors: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) http://lucene.472066.n3.nabble.com/file/n4145519/Screenshot_794.png Solr version is 4.8.1, on Ubuntu Linux. We have 2 nodes, one run 2 shards and another 2 replicas. Errors happen during indexing process. Does it require some tweaks/optimizations? I have no idea where to look to fix this. Any suggestions are welcome. Thank you, Alex -- View this message in context: http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: java.net.SocketException: Connection reset
On 7/7/2014 7:30 AM, heaven wrote: Yeah. the heap is huge, need to optimize the caches. It was 8Gb previously, had to increase because there were out of memory errors. Using ConcMarkSweepGC, which is supposed to not lock the world. At one time the only thing I was using that was non-default for garbage collection was turning on CMS. I had VERY bad GC pauses. I use the following settings now, and my pause problems are much better. There is very likely some other combination of settings that would do even better with Solr: http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Thanks, Shawn
Re: java.net.SocketException: Connection reset
Today this had happened again + this one: null:java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) at java.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181) at org.apache.http.impl.io.ChunkedOutputStream.flushCache(ChunkedOutputStream.java:111) at org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:193) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner$1.writeTo(ConcurrentUpdateSolrServer.java:206) at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:69) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265) at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Previously we had all 4 instances on a single node so I thought these errors might be result of high load. Like if some request taking too long to complete or something like that. And we always had missing docs in the index or vise verse some docs remains in the index when they shouldn't (even though it is supposed to recover from the log and our index queue never remove docs from it until it gets a successful response from Solr). But now we run shards and replicas on separate nodes with lots of resources and a very fast disk storage. And it still causes weird errors. It seems Solr is buggy as hell, that's my impression after a few years of usage. And it doesn't get better in this aspect, these errors follow us from the very beginning. -- View this message in context: http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519p4145675.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: java.net.SocketException: Connection reset
What's the %system load on your nodes? What servlet container are you using? Are you writing a single document per update, or in batches? How many clients are attached to your cloud? Michael Della Bitta Applications Developer o: +1 646 532 3062 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY 10017 t: @appinions https://twitter.com/Appinions | g+: plus.google.com/appinions https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts w: appinions.com http://www.appinions.com/ On Thu, Jul 3, 2014 at 2:06 PM, heaven aheave...@gmail.com wrote: Hi, trying DigitalOcean for Solr, everything seems well, except sometimes I see these errors: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:196) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:233) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) http://lucene.472066.n3.nabble.com/file/n4145519/Screenshot_794.png Solr version is 4.8.1, on Ubuntu Linux. We have 2 nodes, one run 2 shards and another 2 replicas. Errors happen during indexing process. Does it require some tweaks/optimizations? I have no idea where to look to fix this. Any suggestions are welcome. Thank you, Alex -- View this message in context: http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: java.net.SocketException: Connection reset
Hello, usually the loading is not high at all: http://lucene.472066.n3.nabble.com/file/n4145550/Screenshot_795.png We're using bundled jetty and writing in batches by 50-100 documents and only using soft and auto commits. About clients, we have 4 processes and each could run up to 5 threads. At the time when those errors happened only 2 of those processes (queues) were launched. So the number of clients was not higher that 10 at a time. Best, Alex -- View this message in context: http://lucene.472066.n3.nabble.com/java-net-SocketException-Connection-reset-tp4145519p4145550.html Sent from the Solr - User mailing list archive at Nabble.com.
