Re: JVM OOM when using field collapse component
heap space is 4gb set to grow up to 8gb, usage is normally ~1-2gb, seems to happen within a few searches. if its just me ill try to isolate it, it could be some other part of my implementation thx much On Fri, Oct 2, 2009 at 1:18 AM, Martijn v Groningen wrote: > No I have not encountered OOM exception yet with current field collapse patch. > How large is your configured JVM heap space (-Xmx)? Field collapsing > requires more memory then regular searches so. Does Solr run out of > memory during the first search(es) or does it run out of memory after > a while when it performed quite a few field collapse searches? > > I see that you are also using the collapse.includeCollapsedDocs.fl > parameter for your search. This feature will require more memory then > a normal field collapse search. > > I normally give the Solr instance a heap space of 1024M when having an > index of a few million. > > Martijn > > 2009/10/2 Joe Calderon : >> i gotten two different out of memory errors while using the field >> collapsing component, using the latest patch (2009-09-26) and the >> latest nightly, >> >> has anyone else encountered similar problems? my collection is 5 >> million results but ive gotten the error collapsing as little as a few >> thousand >> >> SEVERE: java.lang.OutOfMemoryError: Java heap space >> at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173) >> at >> org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749) >> at >> org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757) >> at >> org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292) >> at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233) >> at >> org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402) >> at >> org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115) >> at >> org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208) >> at >> org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) >> at >> org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148) >> at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387) >> at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) >> at >> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> at >> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) >> at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> at org.mortbay.jetty.Server.handle(Server.java:326) >> at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539) >> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) >> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) >> >> SEVERE: java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.solr.util.DocSetScoreCollector.(DocSetScoreCollector.java:44) >> at >> org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68) >> at >> org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205) >> at >> org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) >> at >> org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) >> at >> org.
Re: JVM OOM when using field collapse component
No I have not encountered OOM exception yet with current field collapse patch. How large is your configured JVM heap space (-Xmx)? Field collapsing requires more memory then regular searches so. Does Solr run out of memory during the first search(es) or does it run out of memory after a while when it performed quite a few field collapse searches? I see that you are also using the collapse.includeCollapsedDocs.fl parameter for your search. This feature will require more memory then a normal field collapse search. I normally give the Solr instance a heap space of 1024M when having an index of a few million. Martijn 2009/10/2 Joe Calderon : > i gotten two different out of memory errors while using the field > collapsing component, using the latest patch (2009-09-26) and the > latest nightly, > > has anyone else encountered similar problems? my collection is 5 > million results but ive gotten the error collapsing as little as a few > thousand > > SEVERE: java.lang.OutOfMemoryError: Java heap space > at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173) > at > org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749) > at > org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757) > at > org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292) > at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233) > at > org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402) > at > org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115) > at > org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208) > at > org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) > at > org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) > > SEVERE: java.lang.OutOfMemoryError: Java heap space > at > org.apache.solr.util.DocSetScoreCollector.(DocSetScoreCollector.java:44) > at > org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68) > at > org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205) > at > org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) > at > org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) > at > org.mortbay.jetty
JVM OOM when using field collapse component
i gotten two different out of memory errors while using the field collapsing component, using the latest patch (2009-09-26) and the latest nightly, has anyone else encountered similar problems? my collection is 5 million results but ive gotten the error collapsing as little as a few thousand SEVERE: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:173) at org.apache.lucene.util.OpenBitSet.ensureCapacityWords(OpenBitSet.java:749) at org.apache.lucene.util.OpenBitSet.ensureCapacity(OpenBitSet.java:757) at org.apache.lucene.util.OpenBitSet.expandingWordNum(OpenBitSet.java:292) at org.apache.lucene.util.OpenBitSet.set(OpenBitSet.java:233) at org.apache.solr.search.AbstractDocumentCollapser.addCollapsedDoc(AbstractDocumentCollapser.java:402) at org.apache.solr.search.NonAdjacentDocumentCollapser.doCollapsing(NonAdjacentDocumentCollapser.java:115) at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:208) at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520) SEVERE: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.util.DocSetScoreCollector.(DocSetScoreCollector.java:44) at org.apache.solr.search.NonAdjacentDocumentCollapser.doQuery(NonAdjacentDocumentCollapser.java:68) at org.apache.solr.search.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:205) at org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:98) at org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:66) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)