Re: solr stuck in writing to inexisting sockets
Hi, Yes, sometimes it takes 5 minutes for a query. I agree this is not desirable. However, if the application has no control over the input queries other that closing the socket after a while, solr should not continue writing the response, but terminate the thread. In general, is there a way to quantify the complexity of a given query on a certain index? Some general guidelines which can be used by non-technical people? Thanks a lot, roxana --- On Sun, 10/31/10, Erick Erickson erickerick...@gmail.com wrote: From: Erick Erickson erickerick...@gmail.com Subject: Re: solr stuck in writing to inexisting sockets To: solr-user@lucene.apache.org Date: Sunday, October 31, 2010, 2:29 AM Are you saying that your Solr server is at times taking 5 minutes to complete? If so, I'd get to the bottom of that first off. My first guess would be you're either hitting memory issues and swapping horribly or..well, that would be my first guess. Best Erick On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta anghelu...@yahoo.comwrote: Hi all, We are using Solr over Jetty with a large index, sharded and distributed over multiple machines. Our queries are quite long, involving boolean and proximity operators. We cut the connection at the client side after 5 minutes. Also, we are using parameter timeAllowed to stop executing it on the server after a while. We quite often run into situations when solr blocks. The load on the server increases and a thread dump on the solr process shows many threads like below: btpool0-49 prio=10 tid=0x7f73afe1d000 nid=0x3581 runnable [0x451a] java.lang.Thread.State: RUNNABLE at java.io.PrintWriter.write(PrintWriter.java:362) at org.apache.solr.common.util.XML.escape(XML.java:206) at org.apache.solr.common.util.XML.escapeCharData(XML.java:79) at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832) at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564) at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435) at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514) at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485) at org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588) at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130) at org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) .. A netstat on the machine shows sockets in state CLOSE_WAIT. However, they are fewer than the number of RUNNABLE threads as the above. Why is this happening? Is there anything we can do to avoid getting in these situations? Thanks, roxana
Re: solr stuck in writing to inexisting sockets
I'm going to nudge you in the direction of understanding why the queries take so long in the first place rather than going toward the blunt approach of cutting them off after some time. The fact that you don't control the queries submitted doesn't prevent you from trying to understand what is taking so long. The first thing I'd look for is whether the system is memory starved. What JVM are you using and what memory parameters are you giving it? What version of Solr are you using? Have you tried any performance monitoring to determine what is happening? The reason I'm pushing in this direction is that 5 minute searches are pathological. Once you're up in that range, virtually any fix you come up with will simply mask the underlying problems, and you'll be forever chasing the next manifestation of the underlying problem. Besides, I don't know how you'd stop Solr processing a query mid-way through, I don't know of any way to make that happen. Best Erick On Mon, Nov 1, 2010 at 9:30 AM, Roxana Angheluta anghelu...@yahoo.comwrote: Hi, Yes, sometimes it takes 5 minutes for a query. I agree this is not desirable. However, if the application has no control over the input queries other that closing the socket after a while, solr should not continue writing the response, but terminate the thread. In general, is there a way to quantify the complexity of a given query on a certain index? Some general guidelines which can be used by non-technical people? Thanks a lot, roxana --- On Sun, 10/31/10, Erick Erickson erickerick...@gmail.com wrote: From: Erick Erickson erickerick...@gmail.com Subject: Re: solr stuck in writing to inexisting sockets To: solr-user@lucene.apache.org Date: Sunday, October 31, 2010, 2:29 AM Are you saying that your Solr server is at times taking 5 minutes to complete? If so, I'd get to the bottom of that first off. My first guess would be you're either hitting memory issues and swapping horribly or..well, that would be my first guess. Best Erick On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta anghelu...@yahoo.com wrote: Hi all, We are using Solr over Jetty with a large index, sharded and distributed over multiple machines. Our queries are quite long, involving boolean and proximity operators. We cut the connection at the client side after 5 minutes. Also, we are using parameter timeAllowed to stop executing it on the server after a while. We quite often run into situations when solr blocks. The load on the server increases and a thread dump on the solr process shows many threads like below: btpool0-49 prio=10 tid=0x7f73afe1d000 nid=0x3581 runnable [0x451a] java.lang.Thread.State: RUNNABLE at java.io.PrintWriter.write(PrintWriter.java:362) at org.apache.solr.common.util.XML.escape(XML.java:206) at org.apache.solr.common.util.XML.escapeCharData(XML.java:79) at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832) at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564) at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435) at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514) at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485) at org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588) at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130) at org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) .. A netstat on the machine shows sockets in state CLOSE_WAIT. However, they are fewer than the number of RUNNABLE threads as the above. Why is this happening? Is there anything we can do to avoid getting in these situations? Thanks, roxana
Re: solr stuck in writing to inexisting sockets
Besides, I don't know how you'd stop Solr processing a query mid-way through, I don't know of any way to make that happen. The timeAllowed parameter causes a timeout in the Solr server to kill the searching thread. They uses that now. But, yes, Erick is right- there is a fundamental problem you should solve. Since they are all stuck in returning XML results, there is something wrong in reading back results. It is possible that there is a bug in timeAllowed, where the kill-this-thread hits while returning the results and the handler for this does not work correctly when returning results. It would be great if someone wrote a unit test for this (not me) and posted it. On Mon, Nov 1, 2010 at 8:44 AM, Erick Erickson erickerick...@gmail.com wrote: I'm going to nudge you in the direction of understanding why the queries take so long in the first place rather than going toward the blunt approach of cutting them off after some time. The fact that you don't control the queries submitted doesn't prevent you from trying to understand what is taking so long. The first thing I'd look for is whether the system is memory starved. What JVM are you using and what memory parameters are you giving it? What version of Solr are you using? Have you tried any performance monitoring to determine what is happening? The reason I'm pushing in this direction is that 5 minute searches are pathological. Once you're up in that range, virtually any fix you come up with will simply mask the underlying problems, and you'll be forever chasing the next manifestation of the underlying problem. Besides, I don't know how you'd stop Solr processing a query mid-way through, I don't know of any way to make that happen. Best Erick On Mon, Nov 1, 2010 at 9:30 AM, Roxana Angheluta anghelu...@yahoo.comwrote: Hi, Yes, sometimes it takes 5 minutes for a query. I agree this is not desirable. However, if the application has no control over the input queries other that closing the socket after a while, solr should not continue writing the response, but terminate the thread. In general, is there a way to quantify the complexity of a given query on a certain index? Some general guidelines which can be used by non-technical people? Thanks a lot, roxana --- On Sun, 10/31/10, Erick Erickson erickerick...@gmail.com wrote: From: Erick Erickson erickerick...@gmail.com Subject: Re: solr stuck in writing to inexisting sockets To: solr-user@lucene.apache.org Date: Sunday, October 31, 2010, 2:29 AM Are you saying that your Solr server is at times taking 5 minutes to complete? If so, I'd get to the bottom of that first off. My first guess would be you're either hitting memory issues and swapping horribly or..well, that would be my first guess. Best Erick On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta anghelu...@yahoo.com wrote: Hi all, We are using Solr over Jetty with a large index, sharded and distributed over multiple machines. Our queries are quite long, involving boolean and proximity operators. We cut the connection at the client side after 5 minutes. Also, we are using parameter timeAllowed to stop executing it on the server after a while. We quite often run into situations when solr blocks. The load on the server increases and a thread dump on the solr process shows many threads like below: btpool0-49 prio=10 tid=0x7f73afe1d000 nid=0x3581 runnable [0x451a] java.lang.Thread.State: RUNNABLE at java.io.PrintWriter.write(PrintWriter.java:362) at org.apache.solr.common.util.XML.escape(XML.java:206) at org.apache.solr.common.util.XML.escapeCharData(XML.java:79) at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832) at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564) at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435) at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514) at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485) at org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588) at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130) at org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java
Re: solr stuck in writing to inexisting sockets
Are you saying that your Solr server is at times taking 5 minutes to complete? If so, I'd get to the bottom of that first off. My first guess would be you're either hitting memory issues and swapping horribly or..well, that would be my first guess. Best Erick On Thu, Oct 28, 2010 at 5:23 AM, Roxana Angheluta anghelu...@yahoo.comwrote: Hi all, We are using Solr over Jetty with a large index, sharded and distributed over multiple machines. Our queries are quite long, involving boolean and proximity operators. We cut the connection at the client side after 5 minutes. Also, we are using parameter timeAllowed to stop executing it on the server after a while. We quite often run into situations when solr blocks. The load on the server increases and a thread dump on the solr process shows many threads like below: btpool0-49 prio=10 tid=0x7f73afe1d000 nid=0x3581 runnable [0x451a] java.lang.Thread.State: RUNNABLE at java.io.PrintWriter.write(PrintWriter.java:362) at org.apache.solr.common.util.XML.escape(XML.java:206) at org.apache.solr.common.util.XML.escapeCharData(XML.java:79) at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832) at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564) at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435) at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514) at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485) at org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588) at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130) at org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) .. A netstat on the machine shows sockets in state CLOSE_WAIT. However, they are fewer than the number of RUNNABLE threads as the above. Why is this happening? Is there anything we can do to avoid getting in these situations? Thanks, roxana
solr stuck in writing to inexisting sockets
Hi all, We are using Solr over Jetty with a large index, sharded and distributed over multiple machines. Our queries are quite long, involving boolean and proximity operators. We cut the connection at the client side after 5 minutes. Also, we are using parameter timeAllowed to stop executing it on the server after a while. We quite often run into situations when solr blocks. The load on the server increases and a thread dump on the solr process shows many threads like below: btpool0-49 prio=10 tid=0x7f73afe1d000 nid=0x3581 runnable [0x451a] java.lang.Thread.State: RUNNABLE at java.io.PrintWriter.write(PrintWriter.java:362) at org.apache.solr.common.util.XML.escape(XML.java:206) at org.apache.solr.common.util.XML.escapeCharData(XML.java:79) at org.apache.solr.request.XMLWriter.writePrim(XMLWriter.java:832) at org.apache.solr.request.XMLWriter.writeStr(XMLWriter.java:684) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:564) at org.apache.solr.request.XMLWriter.writeDoc(XMLWriter.java:435) at org.apache.solr.request.XMLWriter$2.writeDocs(XMLWriter.java:514) at org.apache.solr.request.XMLWriter.writeDocuments(XMLWriter.java:485) at org.apache.solr.request.XMLWriter.writeSolrDocumentList(XMLWriter.java:494) at org.apache.solr.request.XMLWriter.writeVal(XMLWriter.java:588) at org.apache.solr.request.XMLWriter.writeResponse(XMLWriter.java:130) at org.apache.solr.request.XMLResponseWriter.write(XMLResponseWriter.java:34) at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:325) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) .. A netstat on the machine shows sockets in state CLOSE_WAIT. However, they are fewer than the number of RUNNABLE threads as the above. Why is this happening? Is there anything we can do to avoid getting in these situations? Thanks, roxana
