Autocommit
Hello UG I already posted a while ago a problem that one of the solr threads starts using 100% of one of the processor cores on a 4 core system. This doesn't happen right after the start it slightly increaes for about a week until the process runs constantly at 100%. I couldn't figure out a solution for this. I could live with this problem but I think it has an side effect. While the processor load increases the time between two autocommits increases as well. Currently autocommit is set to 3 minutes. After 4 weeks the commits run only every 40 minutes. I have the following solr version installed: Solr Specification Version: 1.2.0 Solr Implementation Version: 1.2.0 - Yonik - 2007-06-02 17:35:12 Lucene Specification Version: 2007-05-20_00-04-53 Lucene Implementation Version: build 2007-05-20 Does anyone have a hint what I could look for? Thanks Michael -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Sorting problem
Hi UG, I just installed the latest nightly solr build (1.2.2007.11.06.08.06.05). I get an exception when I do descending relevancy sorting. Ascending relevancy sorting works fine and sorting on all other fields as well. http://localhost:8080/solr/select/?q=title%3Atestsort=score%20desc Nov 7, 2007 1:13:36 AM org.apache.solr.common.SolrException log SEVERE: java.lang.NullPointerException at org.apache.solr.search.QParser.getSort(QParser.java:158) at org.apache.solr.search.OldLuceneQParser.getSort(LuceneQParserPlugin.java:124) at org.apache.solr.handler.StandardRequestHandler.handleRequestBody(StandardRequestHandler.java:85) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:117) at org.apache.solr.core.SolrCore.execute(SolrCore.java:811) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:206) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:174) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:634) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:445) at java.lang.Thread.run(Thread.java:595) Nov 7, 2007 1:13:36 AM org.apache.solr.core.SolrCore execute INFO: /select/ sort=score+descq=title:test 0 2 Nov 7, 2007 1:13:36 AM org.apache.solr.common.SolrException log SEVERE: java.lang.NullPointerException at org.apache.solr.search.QParser.getSort(QParser.java:158) at org.apache.solr.search.OldLuceneQParser.getSort(LuceneQParserPlugin.java:124) at org.apache.solr.handler.StandardRequestHandler.handleRequestBody(StandardRequestHandler.java:85) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:117) at org.apache.solr.core.SolrCore.execute(SolrCore.java:811) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:206) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:174) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:634) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:445) at java.lang.Thread.run(Thread.java:595) http://localhost:8080/solr/select/?q=title%3Atestsort=score%20asc Nov 7, 2007 1:13:42 AM org.apache.solr.core.SolrCore execute INFO: /select/ sort=score+ascq=title:test 0 17 Does anyone know what could be the problem? Thanks, Michael -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Re: Processor load
On Wed, 2007-04-07 at 10:37 -0400, Yonik Seeley wrote: On 7/3/07, Michael Thessel [EMAIL PROTECTED] wrote: -- 208973 SEVERE: Error during auto-warming of key:[EMAIL PROTECTED]:java 208974 at org.apache.lucene.search.ConjunctionScorer.init(ConjunctionScorer.java:97) 208975 at The Exception and exception message seem to be missing from the stack trace. Could you check your log and see if they are all like this? -Yonik The problem was the autocommit every 10 s. I increased the autocommit time to 60s and now the process runs only at 100% for 30-40s per minute. Thanks a lot, Michael -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Re: Processor load
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:77) 211064 at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) 211065 at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191) 211066 at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:159) 211067 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterCha 211068 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java: 211069 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:228) 211070 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) 211071 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) 211072 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) 211073 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 211074 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:216) 211075 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) 211076 at org.apache.coyote.http11.Http11Protocol $Http11ConnectionHandler.process(Http11Protoco 211077 at org.apache.tomcat.util.net.JIoEndpoint $Worker.run(JIoEndpoint.java:445) 211078 at java.lang.Thread.run(Thread.java:595) -- The first two errors only appear when I send queries while I'm indexing. The LockObtainFailedException always comes up after 1-2 hours of indexing. I found this patch for the LockObtainFailedException problem. Is this currently the official way of fixing the problem? Is this patch already available with one of the nightly builds? https://issues.apache.org/jira/browse/SOLR-240 Thanks and Cheers, Michael On Thu, 2007-28-06 at 16:31 -0700, Otis Gospodnetic wrote: I think you'll need to add this to your Java command line (but you'll need to kill your JVM first): -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9889 Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share - Original Message From: Mike Klaas [EMAIL PROTECTED] To: solr-user@lucene.apache.org Sent: Friday, June 29, 2007 1:01:03 AM Subject: Re: Processor load On 28-Jun-07, at 10:52 AM, Michael Thessel wrote: Hi Mike Thanks for your quick response. The restarting the server hasn't any effect. Im not a Java expert and not really familiar with the java tools. I'm no expert in this matter either. Have you tried kill -QUIT? (it prints a thread dump to stderr). -Mike Running jconsole on the 100% process (as the same user as the server is running): # jconsole 23248 Unable to attach to 23248: Could not map vmid to user Name Running jps: # jps -mlv 11824 sun.tools.jps.Jps -mlv -Dapplication.home=/var/home/solr/src/jdk1.5.0_11 -Xms8m 23228 org.apache.catalina.startup.Bootstrap start -Xms1024M -Xmx2048M -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/home/solr/tomcat/conf/ logging.properties -Djava.endorsed.dirs=/home/solr/tomcat/endorsed - Dcatalina.base=/home/solr/tomcat -Dcatalina.home=/home/solr/tomcat - Djava.io.tmpdir=/home/solr/tomcat/temp Running jconsole on the parent process: # jconsole 23228 23228 is not a managed VM. Cheers, Michael On Thu, 2007-28-06 at 10:14 -0700, Mike Klaas wrote: On 28-Jun-07, at 9:36 AM, Michael Thessel wrote: Hello UG I got a solr server running to index the posts of a forum. The server is running in a tomcat installation on a 4-processor Opteron server. One of the threads is eating up 100% of one processor all the time. As you can see in the ps output it is a fork of one of the childs: have you tried looking at a thread dump using jConsole/kill -QUIT? -Mike -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806 -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Processor load
=spellcheckerIndexDirspell/str str name=termSourceFieldword/str /requestHandler requestHandler name=/update class=solr.XmlUpdateRequestHandler / requestHandler name=/update/csv class=solr.CSVRequestHandler startup=lazy / requestHandler name=/admin/luke class=org.apache.solr.handler.admin.LukeRequestHandler / requestHandler name=/admin/system class=org.apache.solr.handler.admin.SystemInfoHandler / requestHandler name=/admin/plugins class=org.apache.solr.handler.admin.PluginInfoHandler / requestHandler name=/admin/threads class=org.apache.solr.handler.admin.ThreadDumpHandler / requestHandler name=/admin/properties class=org.apache.solr.handler.admin.PropertiesRequestHandler / requestHandler name=/debug/dump class=solr.DumpRequestHandler lst name=defaults str name=echoParamsexplicit/str !-- for all params (including the default etc) use: 'all' -- str name=echoHandlertrue/str /lst /requestHandler queryResponseWriter name=xslt class=org.apache.solr.request.XSLTResponseWriter int name=xsltCacheLifetimeSeconds5/int /queryResponseWriter admin defaultQuerysolr/defaultQuery gettableFilessolrconfig.xml schema.xml admin-extra.html/gettableFiles pingQuery qt=dismaxamp;q=solramp;start=3amp;fq=id:[* TO *]amp;fq=cat:[* TO *] /pingQuery /admin /config -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
filter query speed
Hello UG, I've got a problem with filtered queries. I have an index with about 8 million documents. I save a timestamp (not the time of indexing) for each document as an integer field. Querying the index is pretty fast. But when I filter on the timestamp the queries are extremely slow, even if the unfiltered search is already cached. schema.xml: ... field name=dateline type=integer indexed=true stored=false/ ... INFO: /select/ rows=25start=0q=((title:(test)+AND+is_starter:true)^8 +OR+pagetext:(test)^6+OR+title_pagetext:(test)^4+);+score+descfl= +score,postidqt=standardstylesheet=version=2.1 0 5 INFO: /select/ rows=25start=0fq=dateline:[0+TO +1181237598]+q=((title:(test)+AND+is_starter:true)^8+OR +pagetext:(test)^6+OR+title_pagetext:(test)^4+);+score+descfl= +score,postidqt=standardstylesheet=version=2.1 0 79495 I currently run version: Solr Specification Version: 1.1.2007.05.24.08.06.21 Solr Implementation Version: nightly - yonik - 2007-05-24 08:06:21 Lucene Specification Version: 2007-05-20_00-04-53 Lucene Implementation Version: build 2007-05-20 Tomcat: 6.0.10 Cheers, Michael -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Re: filter query speed
Hey Yoink, thanks a lot for your quick reply. I suspect that the endpoint to your dateline filter changes often, hence caching is doing no good. Is then endpoint (1181237598) derived from the current time? Yes, it is. If so, there are some things you can do: 1) make it faster to generate a new filter by limiting the number of terms in the dateline field (during indexing, always round it to the nearest day) 2) allow solr to reuse previously generated filters more often by rounding the dateline endpoint during query time. You most likely want to do #2, and probably #1 (depending on how often you commit new changes to the index). I will give both of them a try. Is there a general speed problem with range searches in solr? It looks a bit strange for me, that a query for a term takes 5 ms while adding a filter to the same resultset takes 80s? Cheers, Michael -- Michael Thessel [EMAIL PROTECTED] Gossamer Threads Inc. http://www.gossamer-threads.com/ Tel: (604) 687-5804 Fax: (604) 687-5806
Re: Solr performance warnings
Mike Klaas wrote: On 4/20/07, Michael Thessel [EMAIL PROTECTED] wrote: Hey Erik, thanks for the fast reply. Yes this could be possible. I currently got solr running for the indexing of a forum with 100k users. It could definitely be possible that two commits overlap. But I need to commit all changes because the new posts must be available in the search as soon as they are posted. Do you think there is a way to optimize this? As soon as is a rather vague requirement. If you can specify the minimum acceptible delay, then you can use Solr's autocommit functionality to trigger commits. -Mike I didn't know about the timed commits. That's perfect for me. Thanks, Michael
Re: Solr performance warnings
Michael Thessel wrote: Mike Klaas wrote: On 4/20/07, Michael Thessel [EMAIL PROTECTED] wrote: Hey Erik, thanks for the fast reply. Yes this could be possible. I currently got solr running for the indexing of a forum with 100k users. It could definitely be possible that two commits overlap. But I need to commit all changes because the new posts must be available in the search as soon as they are posted. Do you think there is a way to optimize this? As soon as is a rather vague requirement. If you can specify the minimum acceptible delay, then you can use Solr's autocommit functionality to trigger commits. -Mike I didn't know about the timed commits. That's perfect for me. Thanks, Michael The timed commits don't work for me. The webinterface says 0 commits since the server was restarted. And nothing in the logs as well. I use: apache-solr-1.1.0-incubating My updateHandler section from solrconfig.xml: updateHandler class=solr.DirectUpdateHandler2 autoCommit maxTime1/maxTime /autoCommit /updateHandler I also tried maxTime10/maxTime in case its seconds and not ms. Cheers Michael
Solr performance warnings
Hello, in my logs I get from time to time this message: INFO: PERFORMANCE WARNING: Overlapping onDeckSearchers=2 What does this mean? What can I do to avoid this? Cheers, Michael
