hmm, i think you are actually running out of virtual address space, even on 64-bit!
http://msdn.microsoft.com/en-us/library/aa366778(v=VS.85).aspx#memory_limits Apparently windows limits you to 8TB virtual address space (ridiculous), so i think you should try one of the following: * continue using mmap directory, but specify MMapDirectoryFactory yourself, and specify the maxChunkSize parameter. The default maxChunkSize is Integer.MAX_VALUE, but with a smaller one you might be able to work around fragmentation problems. * continue using mmap directory, but adjust index params such as merge factor. * use SimpleFSDirectory instead: (SimpleFSDirectoryFactory). But the big downside is that its slower and you have no i/o concurrency. separately, it might be a good idea to consider splitting up your 1.1B documents/550GB index across more than one machine... :) On Fri, Dec 24, 2010 at 1:19 AM, Rok Rejc <rokrej...@gmail.com> wrote: > Hi all, > > I have created a new index (using Solr trunk version from 17th December, > running on Windows 7 & Tomcat 6, 64 bit JVM) with around 1.1 billion of > documents (index size around 550GB, mergeFactor=20). > > After the (csv) import I have commited the data and got this error: > > HTTP Status 500 - Severe errors in solr configuration. Check your log files > for more detailed information on what may be wrong. > ------------------------------------------------------------- > java.lang.RuntimeException: java.io.IOException: Map failed at > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1095) at > org.apache.solr.core.SolrCore.<init>(SolrCore.java:587) at > org.apache.solr.core.CoreContainer.create(CoreContainer.java:660) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:412) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:294) at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:243) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) > at > org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:115) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4001) > at org.apache.catalina.core.StandardContext.start(StandardContext.java:4651) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546) at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637) > at > org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563) > at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498) at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at > org.apache.catalina.core.StandardHost.start(StandardHost.java:785) at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445) at > org.apache.catalina.core.StandardService.start(StandardService.java:519) at > org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at > org.apache.catalina.startup.Catalina.start(Catalina.java:581) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at > java.lang.reflect.Method.invoke(Unknown Source) at > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: > java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown > Source) at > org.apache.lucene.store.MMapDirectory$MultiMMapIndexInput.<init>(MMapDirectory.java:327) > at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:209) > at > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:68) > at > org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:208) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:529) at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:504) at > org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:123) at > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:91) at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:623) > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86) at > org.apache.lucene.index.IndexReader.open(IndexReader.java:437) at > org.apache.lucene.index.IndexReader.open(IndexReader.java:316) at > org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1084) ... 33 more > Caused by: java.lang.OutOfMemoryError: Map failed at > sun.nio.ch.FileChannelImpl.map0(Native Method) ... 48 more > > I can see that the error is going down to lucene and java, but I don't have > a clue what should I do... Any suggestions? > > Thanks and merry christmas:) > > Rok >
