Re: Out of memory on Solr sorting

Mark Miller Tue, 22 Jul 2008 12:46:38 -0700

Because to sort efficiently, Solr loads the term to sort on for each docin the index into an array. For ints,longs, etc its just an array thesize of the number of docs in your index (i believe deleted or not). Fora String its an array to hold each unique string and an array of intsindexing into the String array.

So if you do a sort, and search for something that only gets 1 doc as ahit...your still loading up that field cache for every single doc inyour index on the first search. With solr, this happens in thebackground as it warms up the searcher. The end story is, you need moreRAM to accommodate the sort most likely...have you upped your xmxsetting? I think you can roughly say a 2 million doc index would need40-50 MB (depending and rough, but to give an idea) per field yoursorting on.


- Mark

sundar shankar wrote:

Thanks Fuad.
                  But why does just sorting provide an OOM. I executed the 
query without adding the sort clause it executed perfectly. In fact I even 
tried remove the maxrows=10 and executed. it came out fine. Queries with bigger 
results seems to come out fine too. But why just sort of that too just 10 rows??
-Sundar
Date: Tue, 22 Jul 2008 12:24:35 -0700> From: [EMAIL PROTECTED]> To: solr-user@lucene.apache.org> Subject: RE: Out of memory on Solr sorting> > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)> > - this piece of code do not request Array[100M] (as I seen with > Lucene), it asks only few bytes / Kb for a field...> > > Probably 128 - 512 is not enough; it is also advisable to use equal sizes> -Xms1024M -Xmx1024M> (it minimizes GC frequency, and itensures that 1024M is available at startup)> > OOM happens also with fragmented memory, when application requests big > contigues fragment and GC is unable to optimize; looks like your > application requests a little and memory is not available...> > > Quoting sundar shankar <[EMAIL PROTECTED]>:> > >> >> >> >> From: [EMAIL PROTECTED]> >> To: solr-user@lucene.apache.org> >> Subject: Out of memory on Solr sorting> >> Date: Tue, 22 Jul 2008 19:11:02 +0000> >>> >>> >> Hi,> >> Sorry again fellos. I am not sure whats happening. The day with > >> solr is bad for me I guess. EZMLM didnt let me send any mails this > >> morning. Asked me to confirm subscription and when I did, it said I > >> was already a member. Now my mails are all coming out bad. Sorry > >> for troubling y'all this bad. I hope this mail comes out right.> >> >> > Hi,> > We are developing a product in a agile manner and the current > > implementation has a data of size just about a 800 megs in dev.> > The memory allocated to solr on dev (Dual core Linux box) is 128-512.> >> > My config> > =========> >> > > >> > <filterCache> > class="solr.LRUCache"> > size="512"> > initialSize="512"> > autowarmCount="256"/>> >> > <queryResultCache> > class="solr.LRUCache"> > size="512"> > initialSize="512"> > autowarmCount="256"/>> >> > <documentCache> > class="solr.LRUCache"> > size="512"> > initialSize="512"> > autowarmCount="0"/>> >> > <enableLazyFieldLoading>true</enableLazyFieldLoading>> >> >> > My Field> > =======> >> > <fieldType name="autocomplete" class="solr.TextField">> > <analyzer type="index">> > <tokenizer class="solr.KeywordTokenizerFactory"/>> > <filter class="solr.LowerCaseFilterFactory" />> > <filter class="solr.PatternReplaceFilterFactory" > > pattern="([^a-z0-9])" replacement="" replace="all" />> > <filter class="solr.EdgeNGramFilterFactory" > > maxGramSize="100" minGramSize="1" />> > </analyzer>> > <analyzer type="query">> > <tokenizer class="solr.KeywordTokenizerFactory"/>> > <filter class="solr.LowerCaseFilterFactory" />> > <filter class="solr.PatternReplaceFilterFactory" > > pattern="([^a-z0-9])" replacement="" replace="all" />> > <filter class="solr.PatternReplaceFilterFactory" > > pattern="^(.{20})(.*)?" replacement="$1" replace="all" />> > </analyzer>> > </fieldType>> >> >> > Problem> > ======> >> > I execute a query that returns 24 rows of result. I pick 10 out of > > it. I have no problem when I execute this.> > But When I do sort it by a String field that is fetched from this > > result. I get an OOM. I am able to execute several> > other queries with no problem. Just having a sort asc clause added > > to the query throws an OOM. Why is that.> > What should I have ideally done. My config on QA is pretty similar > > to the dev box and probably has more data than on dev.> > It didnt throw any OOM during the integration test. The Autocomplete > > is a new field we added recently.> >> > Another point is that the indexing is done with a field of type string> > <field name="XXX" type="string" indexed="true" stored="true" > > termVectors="true"/>> >> > and the autocomplete field is a copy field.> >> > The sorting is done based on string field.> >> > Please do lemme know what mistake am I doing?> >> > Regards> > Sundar> >> > P.S: The stack trace of the exception is> >> >> > Caused by: org.apache.solr.client.solrj.SolrServerException: Error > > executing query> > at > > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:86)> > at > > org.apache.solr.client.solrj.impl.BaseSolrServer.query(BaseSolrServer.java:101)> > at > > com.apollo.sisaw.solr.service.AbstractSolrSearchService.makeSolrQuery(AbstractSolrSearchService.java:193)> > ... 105 more> > Caused by: org.apache.solr.common.SolrException: Java heap space > > java.lang.OutOfMemoryError: Java heap space> > at > > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:403)> > at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)> > at > > org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)> > at > > org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:416)> > at > > org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:207)> > at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)> > at > > org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168)> > at > > org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56)> > at > > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:907)> > at > > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)> > at > > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)> > at > > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)> > at > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:156)> > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:128)> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1025)> > at > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)> > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)> > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)> > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)> > at > > org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)> > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)> > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)> > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)> > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)> > at > > org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)> > at > > org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)> > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)> > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)> > at > > org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)> > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)> > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)> > at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)> >> > _________________________________________________________________> > Wish to Marry Now? Click Here to Register FREE> > http://www.shaadi.com/registration/user/index.php?ptnr=mhottag> > >
_________________________________________________________________
Missed your favourite programme? Stop surfing TV channels and start planning 
your weekend TV viewing with our comprehensive TV Listing
http://entertainment.in.msn.com/TV/TVListing.aspx

Re: Out of memory on Solr sorting

Reply via email to