Hi I am really frustrated by this problem.
I have built an index of 1.5 billion data records, with a size of about 170GB. It's been optimised and has 12 separate files in the index directory, looking like below: _2.fdt --- 58G _2.fdx --- 80M _2.fnm--- 900bytes _2.si --- 380bytes _2.lucene41_0.doc --- 46G _2_Lucene41_0.pos --- 22G _2_Lucene41_0.tim --- 37G _2_Lucene41_0.tip --- 766MB _2_nrm.cfe --- 139byte _2_nrm.cfs --- 5.7G segments.gen -- 20byte segments_1 --- 68byte It sits on a single server with a memory of 32G allocated to it, using default solr setting that are provided with the solr example in the distrubtion. I started the server ok with 32G memory, but any query other than "q=*:*" fails, with the following out of memory exception: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a value smaller than the current chunk size (2147483647)</str><str name="trace">java.lang.RuntimeException: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a value smaller than the current chunk size (2147483647) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:462) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:290) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a value smaller than the current chunk size (2147483647) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:184) at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:158) at org.apache.lucene.codecs.DocValuesArraySource$ByteValues.<init>(DocValuesArraySource.java:113) at org.apache.lucene.codecs.DocValuesArraySource$ByteValues.newFromInput(DocValuesArraySource.java:135) at org.apache.lucene.codecs.lucene40.values.Ints$IntsReader.loadSource(Ints.java:154) at org.apache.lucene.index.DocValues$SourceCache$DirectSourceCache.load(DocValues.java:791) at org.apache.lucene.index.DocValues.getSource(DocValues.java:129) at org.apache.lucene.search.similarities.TFIDFSimilarity$ExactTFIDFDocScorer.<init>(TFIDFSimilarity.java:782) at org.apache.lucene.search.similarities.TFIDFSimilarity.exactSimScorer(TFIDFSimilarity.java:761) at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:88) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:603) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1435) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1310) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:401) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:410) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269) ... 25 more Caused by: java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:101) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155) at sun.nio.ch.IOUtil.read(IOUtil.java:169) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:612) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176) ... 45 more I am quite new to Solr and probably did something obviously wrong with such a huge index... so my question is what are the possible solutions to fix this? -by increasing more memory? -change solr configuration to optimise for large index? if so what should I be looking into? -solrcloud? Again im complete novice... any tips on configuration... # of shards etc? Many thanks! -- View this message in context: http://lucene.472066.n3.nabble.com/170G-index-1-5-billion-documents-out-of-memory-on-query-tp4042696.html Sent from the Solr - User mailing list archive at Nabble.com.
