Stack trace looks normal - it's just a multi-term query instantiating a bitset. The memory is being taken up somewhere else. How many documents are in your index? Can you get a heap dump or use some other memory profiler to see what's taking up the space?
> if I stop query more then ten minutes, the solr instance will start normally. Maybe queries are piling up in threads before the server is ready to handle them and then trying to handle them all at once gives an OOM? Is this live traffic or a test? How many concurrent requests get sent? -Yonik http://lucidimagination.com On Wed, Aug 8, 2012 at 2:43 AM, Jam Luo <cooljam2...@gmail.com> wrote: > Aug 06, 2012 10:05:55 AM org.apache.solr.common.SolrException log > SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java > heap space > at > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:456) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:284) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) > at org.eclipse.jetty.server.Server.handle(Server.java:351) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) > at > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.OutOfMemoryError: Java heap space > at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:54) > at > org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104) > at > org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:129) > at > org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:318) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:507) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:280) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1394) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1269) > at > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:384) > at > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:420) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:204) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1544) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:499) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) > at org.eclipse.jetty.server.Server.handle(Server.java:351) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900) > > This error often appear at the startup, no data write to the index, > but it have a lot of query request. if I stop query more then ten > minutes, the solr instance will start normally. > My index data in solr data directory is 200g+, RAM is 16g, jvm > properties is > -Xmx10g > -Xss256k > -Xmn512m > -XX:+UseCompressedOops > The OOM and the peer startup fail may be uncorrelated, but this two > things often happen in the same solr instance and the same time. > > I can provide the full log file if you want. > > thanks > > > > > 2012/8/7 Mark Miller <markrmil...@gmail.com> > >> Still no idea on the OOM - please send the stacktrace if you can. >> >> As for doing a replication recovery when it should not be necessary, yonik >> just committed a fix for that a bit ago. >> >> On Aug 7, 2012, at 9:41 AM, Mark Miller <markrmil...@gmail.com> wrote: >> >> > >> > On Aug 7, 2012, at 5:49 AM, Jam Luo <cooljam2...@gmail.com> wrote: >> > >> >> Hi >> >> I have big index data files more then 200g, there are two solr >> >> instance in a shard. leader startup and is ok, but the peer alway OOM >> >> when it startup. >> > >> > Can you share the OOM msg and stacktrace please? >> > >> >> The peer alway download index files from leader because >> >> of recoveringAfterStartup property in RecoveryStrategy, total time >> taken >> >> for download : 2350 secs. if data of the peer is empty, it is ok, but >> the >> >> leader and the peer have a same generation number, why the peer >> >> do recovering? >> > >> > We are looking into this. >> > >> >> >> >> thanks >> >> cooljam >> > >> > - Mark Miller >> > lucidimagination.com >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >> - Mark Miller >> lucidimagination.com >> >> >> >> >> >> >> >> >> >> >> >>
