Re: solr/tomcat stops responding
Hello Kevin, I am also facing same problem After few hours or few day my solr server getting crash. I try to download following patch but its not accessible now. i am using 3.1 version of solr. http://people.apache.org/~yonik/solr/current/solr.war - Regards, Suneel Pandey Sr. Software Developer -- View this message in context: http://lucene.472066.n3.nabble.com/solr-tomcat-stops-responding-tp474577p3998435.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr/tomcat stops responding
You are referring to a very old thread Did you take any heap dump and thread dumo? They can help you get more insight. -Saroj On Tue, Jul 31, 2012 at 9:04 AM, Suneel pandey.sun...@gmail.com wrote: Hello Kevin, I am also facing same problem After few hours or few day my solr server getting crash. I try to download following patch but its not accessible now. i am using 3.1 version of solr. http://people.apache.org/~yonik/solr/current/solr.war - Regards, Suneel Pandey Sr. Software Developer -- View this message in context: http://lucene.472066.n3.nabble.com/solr-tomcat-stops-responding-tp474577p3998435.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr/tomcat stops responding
On 12/2/06, Kevin Lewandowski [EMAIL PROTECTED] wrote: accept connections for 3 or 4 hours ... did you try taking some thread dumps like yonik suggested to see what all the threads were doing? A kill -3 will not kill the process. It does nothing and there's no thread dump on the console. kill -9 does kill it though. Hmmm, on most Linux/UNIX systems, sending the QUIT signal does nothing else but generate a stack trace to the console or a log file. If you don't start tomcat by hand, the stack trace may go somewhere else I suppose. This would be useful to learn how to do on your particular system (and we should add it to a debugging/troubleshooting wiki too). btw, this has been a bigger problem for me because there's a separate hardware issue and the system freezes about every 12 hours. So I have to reboot it. After that I noticed solr not responding. Are you load-balancing at all, or is this your only search server? FYI, I'm looking into something that will help. -Yonik
Re: solr/tomcat stops responding
Hmmm, on most Linux/UNIX systems, sending the QUIT signal does nothing else but generate a stack trace to the console or a log file. If you don't start tomcat by hand, the stack trace may go somewhere else I suppose. This would be useful to learn how to do on your particular system (and we should add it to a debugging/troubleshooting wiki too). Okay, I figured out how to get the thread dump. It was in the tomcat logfile. I'm attaching it here. Are you load-balancing at all, or is this your only search server? FYI, I'm looking into something that will help. I'm load balancing two solr servers. thanks, Kevin thread_dump.txt.gz Description: GNU Zip compressed data
Re: solr/tomcat stops responding
On 12/2/06, Kevin Lewandowski [EMAIL PROTECTED] wrote: I've done a temporary fix for this by running a proxy in front of tomcat. Then I updated my system startup to start solr, wait 20 seconds, run a few queries, wait 20 seconds, then start the proxy. This is working fine now. But I'd still like to fix the real problem. OK, this may fix it: https://issues.apache.org/jira/browse/SOLR-77 A war with this patch included is here: http://people.apache.org/~yonik/solr/current/solr.war You also need to configure some queries to be done on the firstSearcher event in solrconfig.xml. Uncomment and customize the example one included. A single query should do, just make sure to include a sort that includes *all* of the fields that are commonly sorted on (the new lucene build with the sort fix hasn't been included yet). If you use any filters (fq param) or faceted browsing commands, then include those too. This will cause all searchers to block and wait for initial warming to be completed. If the load balancer is sending traffic during this time, these requests will pile up for a short time, but should complete OK after that. -Yonik
Re: solr/tomcat stops responding
On 12/3/06, Kevin Lewandowski [EMAIL PROTECTED] wrote: Okay, I figured out how to get the thread dump. It was in the tomcat logfile. I'm attaching it here. Nothing sticks out to me as problematic, except for the 150 concurrent requests in progress! Perhaps the real problem already passed, and the result is the pileup of all these requests which thrash eachother and have a hard time making progress. Also throw the memory consumed by all these threads might cause GC to churn, also lowering the amount of CPU available to complete them. Hopefully the patch I just provided, along with warming, will prevent getting into this situation. -Yonik
Re: solr/tomcat stops responding
: Okay, I did that. Shut off traffic to the server, restarted solr, ran : a few queries against it, then turned traffic back on, and it's : running fine now. So maybe the initial flood of requests has something : to do with it? an initial flood of requests to a completley cold index can definitely hang while the index gets loaded, pages get swapped in, and FieldCaches get constructed ... but i'm really amazed that it just flat out wouldn't accept connections for 3 or 4 hours ... did you try taking some thread dumps like yonik suggested to see what all the threads were doing? -Hoss
Re: solr/tomcat stops responding
accept connections for 3 or 4 hours ... did you try taking some thread dumps like yonik suggested to see what all the threads were doing? A kill -3 will not kill the process. It does nothing and there's no thread dump on the console. kill -9 does kill it though. btw, this has been a bigger problem for me because there's a separate hardware issue and the system freezes about every 12 hours. So I have to reboot it. After that I noticed solr not responding. I've done a temporary fix for this by running a proxy in front of tomcat. Then I updated my system startup to start solr, wait 20 seconds, run a few queries, wait 20 seconds, then start the proxy. This is working fine now. But I'd still like to fix the real problem. Let me know if there's anything else I can test or information I can provide. thanks, Kevin
Re: solr/tomcat stops responding
On 12/1/06, Kevin Lewandowski [EMAIL PROTECTED] wrote: My solr installation has been running fine for a few weeks but now after a server reboot it starts and runs for a few seconds, then stops responding. I don't see any errors in the logfiles, apart from snapinstaller not being able to issue a commit. Also, the process is using 100% cpu and stops responding to http requests (admin interface and queries). - check what got changed after the server reboot... anything? - turn on GC logging to see if you are running out of memory and the GC system is taking up the CPU: -Xloggc:gc.log - get a JVM thread dump via kill -3 (assuming Linux/Unix) - try rebuilding the index to see if it's something funky there Is there anything in solr that would cause this? Yes, if it's normally a heavily loaded server that was restarted, there can be a problem of too many search requests hitting it all at once. Many of the requests may want to sort by the same field, and all of them start generating the same fieldcache entry or filter entries at the same time. This causes a lot of memory to quickly be allocated, GC starts churning trying to free up memory, more threads pile up, etc... Part of the fix for this has recently been committed into Lucene (multiple threads won't generate the same FieldCache entry). We could also do more in Solr by not releasing the first searcher until it has been warmed by the static solrconfig.xml entries. To see if this is your problem, restart the server and make sure no traffic goes to it. Then run some queries of the same type that will be hitting it to warm it up, then turn on normal traffic. -Yonik
Re: solr/tomcat stops responding
My solr installation has been running fine for a few weeks but now after a server reboot it starts and runs for a few seconds, then stops responding. I don't see any errors in the logfiles, apart from snapinstaller not being able to issue a commit. Also, the process is using 100% cpu and stops responding to http requests (admin interface and queries). Okay, some more happened after I sent this email. About 3 hours after the reboot solr started running normally again. Then I rebooted it to see if I could reproduce it. This time solr remained in the not-responding state for about 4 hours but I did not wait longer to see if it would come back. - check what got changed after the server reboot... anything? Nothing had been changed on the server. Part of the fix for this has recently been committed into Lucene (multiple threads won't generate the same FieldCache entry). Has that been added to solr yet? I'm running solr-2006-11-20. To see if this is your problem, restart the server and make sure no traffic goes to it. Then run some queries of the same type that will be hitting it to warm it up, then turn on normal traffic. Okay, I did that. Shut off traffic to the server, restarted solr, ran a few queries against it, then turned traffic back on, and it's running fine now. So maybe the initial flood of requests has something to do with it? thanks, Kevin