We have a solr core with about 115 million documents. We are trying to
migrate data and running a simple query with *:* query and with start
and rows param.
The performance is becoming too slow in solr, its taking almost 2 mins
to get 4000 rows and migration is being just too slow. Logs snippet below:
INFO: [coreName] webapp=/solr path=/select
params={start=55438000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=168308
INFO: [coreName] webapp=/solr path=/select
params={start=55446000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=122771
INFO: [coreName] webapp=/solr path=/select
params={start=55454000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=137615
INFO: [coreName] webapp=/solr path=/select
params={start=55450000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=141223
INFO: [coreName] webapp=/solr path=/select
params={start=55462000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=97474
INFO: [coreName] webapp=/solr path=/select
params={start=55458000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=98115
INFO: [coreName] webapp=/solr path=/select
params={start=55466000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=143822
INFO: [coreName] webapp=/solr path=/select
params={start=55474000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=118066
INFO: [coreName] webapp=/solr path=/select
params={start=55470000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=121498
INFO: [coreName] webapp=/solr path=/select
params={start=55482000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=164062
INFO: [coreName] webapp=/solr path=/select
params={start=55478000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=165518
INFO: [coreName] webapp=/solr path=/select
params={start=55486000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=118163
INFO: [coreName] webapp=/solr path=/select
params={start=55494000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=141642
INFO: [coreName] webapp=/solr path=/select
params={start=55490000&q=*:*&wt=javabin&version=2&rows=4000}
hits=115760479 status=0 QTime=145037
I've taken some thread dumps in the solr server and most of the time the
threads seem to be busy in the following stacks mostly:
Is there anything that can be done to improve the performance? Is it a
known issue? Its very surprising that querying for some just rows
starting at some points is taking in order of minutes.
"395883378@qtp-162198005-7" prio=10 tid=0x00007f4aa0636000 nid=0x295a
runnable [0x00007f42865dd000]
java.lang.Thread.State: RUNNABLE
at
org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:252)
at org.apache.lucene.util.PriorityQueue.pop(PriorityQueue.java:184)
at
org.apache.lucene.search.TopDocsCollector.populateResults(TopDocsCollector.java:61)
at
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:156)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1499)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1366)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:457)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:410)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
"1154127582@qtp-162198005-3" prio=10 tid=0x00007f4aa0613800 nid=0x2956
runnable [0x00007f42869e1000]
java.lang.Thread.State: RUNNABLE
at
org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:252)
at
org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:210)
at
org.apache.lucene.search.TopScoreDocCollector$InOrderTopScoreDocCollector.collect(TopScoreDocCollector.java:62)
at org.apache.lucene.search.Scorer.score(Scorer.java:64)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:605)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1491)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1366)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:457)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:410)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
--
---------
Cheers,
Abhishek
