Hi folks, I ran into problem today where I am no longer able to execute any
queries :( due to Out of Memory issues.
I am in the process of investigating the use of different mergeFactors, or
even different merge policies all together.
My question is if I have many segments (i.e. smaller sized segments), will
that also reduce the total memory in RAM required for searching? (my System
is currently allocated 8GB ram and has a ~255GB index). (I'm not fully up
on the 'default merge policy' but I believe with a mergeFactor of 10, that
would mean each segment should be approaching about 25Gb? with ~543 million
documents
of note: this is all running on 1 server.
As seen below.
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.search.cache.LongValuesCreator.fillLongValues(LongValuesCreator.java:141)
at
org.apache.lucene.search.cache.LongValuesCreator.validate(LongValuesCreator.java:84)
at
org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:74)
at
org.apache.lucene.search.cache.LongValuesCreator.create(LongValuesCreator.java:37)
at
org.apache.lucene.search.FieldCacheImpl$Cache.createValue(FieldCacheImpl.java:155)
at
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:188)
at
org.apache.lucene.search.FieldCacheImpl.getLongs(FieldCacheImpl.java:337)
at
org.apache.lucene.search.FieldComparator$LongComparator.setNextReader(FieldComparator.java:504)
at
org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:97)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:207)
at org.apache.lucene.search.Searcher.search(Searcher.java:101)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1389)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1285)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:344)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:273)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:210)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1324)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.openmarket.servletfilters.LogToCSVFilter.doFilter(LogToCSVFilter.java:89)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.openmarket.servletfilters.GZipAutoDeflateFilter.doFilter(GZipAutoDeflateFilter.java:66)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
...etc
--
View this message in context:
http://lucene.472066.n3.nabble.com/Segments-and-Memory-Correlate-tp2694747p2694747.html
Sent from the Solr - User mailing list archive at Nabble.com.
