Re: Map failed at getSearcher
ok i opened https://issues.apache.org/jira/browse/LUCENE-2832 On Fri, Dec 24, 2010 at 12:44 PM, Robert Muir wrote: > On Fri, Dec 24, 2010 at 12:28 PM, Yonik Seeley > wrote: >> On Fri, Dec 24, 2010 at 10:23 AM, Robert Muir wrote: >>> hmm, i think you are actually running out of virtual address space, >>> even on 64-bit! >> >> I don't know if there are any x86 processors that allow 64 bits of >> address space yet. >> AFAIK, they are mostly 48 bit. > > right but 128TB (linux/osx/solarisx86 et al) I think is a worlds of > difference from Windows' 44-bit view (8TB) > >> >> Hmmm, maybe we should default to a smaller value? Perhaps something >> like 1G wouldn't impact performance, but could help avoid OOM due to >> fragmentation? >> > > We already conditionalize the default value... if it would actually > help, I think this could be a good idea, but maybe only for windows > (44bit)? >
Re: Map failed at getSearcher
On Fri, Dec 24, 2010 at 12:28 PM, Yonik Seeley wrote: > On Fri, Dec 24, 2010 at 10:23 AM, Robert Muir wrote: >> hmm, i think you are actually running out of virtual address space, >> even on 64-bit! > > I don't know if there are any x86 processors that allow 64 bits of > address space yet. > AFAIK, they are mostly 48 bit. right but 128TB (linux/osx/solarisx86 et al) I think is a worlds of difference from Windows' 44-bit view (8TB) > > Hmmm, maybe we should default to a smaller value? Perhaps something > like 1G wouldn't impact performance, but could help avoid OOM due to > fragmentation? > We already conditionalize the default value... if it would actually help, I think this could be a good idea, but maybe only for windows (44bit)?
Re: Map failed at getSearcher
On Fri, Dec 24, 2010 at 10:23 AM, Robert Muir wrote: > hmm, i think you are actually running out of virtual address space, > even on 64-bit! I don't know if there are any x86 processors that allow 64 bits of address space yet. AFAIK, they are mostly 48 bit. > http://msdn.microsoft.com/en-us/library/aa366778(v=VS.85).aspx#memory_limits > > Apparently windows limits you to 8TB virtual address space > (ridiculous), so i think you should try one of the following: > * continue using mmap directory, but specify MMapDirectoryFactory > yourself, and specify the maxChunkSize parameter. The default > maxChunkSize is Integer.MAX_VALUE, but with a smaller one you might be > able to work around fragmentation problems. Hmmm, maybe we should default to a smaller value? Perhaps something like 1G wouldn't impact performance, but could help avoid OOM due to fragmentation? -Yonik http://www.lucidimagination.com
Re: Map failed at getSearcher
hmm, i think you are actually running out of virtual address space, even on 64-bit! http://msdn.microsoft.com/en-us/library/aa366778(v=VS.85).aspx#memory_limits Apparently windows limits you to 8TB virtual address space (ridiculous), so i think you should try one of the following: * continue using mmap directory, but specify MMapDirectoryFactory yourself, and specify the maxChunkSize parameter. The default maxChunkSize is Integer.MAX_VALUE, but with a smaller one you might be able to work around fragmentation problems. * continue using mmap directory, but adjust index params such as merge factor. * use SimpleFSDirectory instead: (SimpleFSDirectoryFactory). But the big downside is that its slower and you have no i/o concurrency. separately, it might be a good idea to consider splitting up your 1.1B documents/550GB index across more than one machine... :) On Fri, Dec 24, 2010 at 1:19 AM, Rok Rejc wrote: > Hi all, > > I have created a new index (using Solr trunk version from 17th December, > running on Windows 7 & Tomcat 6, 64 bit JVM) with around 1.1 billion of > documents (index size around 550GB, mergeFactor=20). > > After the (csv) import I have commited the data and got this error: > > HTTP Status 500 - Severe errors in solr configuration. Check your log files > for more detailed information on what may be wrong. > - > java.lang.RuntimeException: java.io.IOException: Map failed at > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1095) at > org.apache.solr.core.SolrCore.(SolrCore.java:587) at > org.apache.solr.core.CoreContainer.create(CoreContainer.java:660) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:412) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:294) at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:243) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4001) > at org.apache.catalina.core.StandardContext.start(StandardContext.java:4651) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546) at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637) > at > org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563) > at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498) at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at > org.apache.catalina.core.StandardHost.start(StandardHost.java:785) at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445) at > org.apache.catalina.core.StandardService.start(StandardService.java:519) at > org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at > org.apache.catalina.startup.Catalina.start(Catalina.java:581) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at > java.lang.reflect.Method.invoke(Unknown Source) at > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: > java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown > Source) at > org.apache.lucene.store.MMapDirectory$MultiMMapIndexInput.(MMapDirectory.java:327) > at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:209) > at > org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:68) > at > org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:208) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:529) at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:504) at > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:123) at > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:91) at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:623) > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86) at > org.apache.lucene.index.IndexReader.op
Re: Map failed at getSearcher
At root, it's an OOM: Caused by: java.lang.OutOfMemoryError: Map failed at I'm guessing that you're optimizing after the import? What are the JVM settings you're using? The standard response is "increase the amount of memory available to the JVM", but it's expensive to change it and only find out you're running over the limit *after* a billion docs. The "standard" advice is to allow the JVM about half the memory available on the machine, leaving the rest for the op system to use as it sees fit, but that's just a starting point. Hope that helps Erick On Fri, Dec 24, 2010 at 1:19 AM, Rok Rejc wrote: > Hi all, > > I have created a new index (using Solr trunk version from 17th December, > running on Windows 7 & Tomcat 6, 64 bit JVM) with around 1.1 billion of > documents (index size around 550GB, mergeFactor=20). > > After the (csv) import I have commited the data and got this error: > > HTTP Status 500 - Severe errors in solr configuration. Check your log files > for more detailed information on what may be wrong. > - > java.lang.RuntimeException: java.io.IOException: Map failed at > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1095) at > org.apache.solr.core.SolrCore.(SolrCore.java:587) at > org.apache.solr.core.CoreContainer.create(CoreContainer.java:660) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:412) at > org.apache.solr.core.CoreContainer.load(CoreContainer.java:294) at > > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:243) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86) > at > > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) > at > > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) > at > > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) > at > > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4001) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4651) > at > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546) at > > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637) > at > > org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563) > at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498) > at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) > at > > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at > org.apache.catalina.core.StandardHost.start(StandardHost.java:785) at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445) at > org.apache.catalina.core.StandardService.start(StandardService.java:519) at > org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at > org.apache.catalina.startup.Catalina.start(Catalina.java:581) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at > java.lang.reflect.Method.invoke(Unknown Source) at > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: > java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown > Source) at > > org.apache.lucene.store.MMapDirectory$MultiMMapIndexInput.(MMapDirectory.java:327) > at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:209) > at > > org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:68) > at > > org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:208) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:529) at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:504) at > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:123) at > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:91) > at > > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:623) > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86) at > org.apache.lucene.index.IndexReader.open(IndexReader.java:437) at > org.apache.lucene.index.IndexReader.open(IndexReader.java:316) at > > org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1084) ... 33 > more >
Map failed at getSearcher
Hi all, I have created a new index (using Solr trunk version from 17th December, running on Windows 7 & Tomcat 6, 64 bit JVM) with around 1.1 billion of documents (index size around 550GB, mergeFactor=20). After the (csv) import I have commited the data and got this error: HTTP Status 500 - Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. - java.lang.RuntimeException: java.io.IOException: Map failed at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1095) at org.apache.solr.core.SolrCore.(SolrCore.java:587) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:660) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:412) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:294) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:243) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:86) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4001) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4651) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:637) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:563) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:498) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:785) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445) at org.apache.catalina.core.StandardService.start(StandardService.java:519) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:581) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(Unknown Source) at org.apache.lucene.store.MMapDirectory$MultiMMapIndexInput.(MMapDirectory.java:327) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:209) at org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:68) at org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:208) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:529) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:504) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:123) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:91) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:623) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86) at org.apache.lucene.index.IndexReader.open(IndexReader.java:437) at org.apache.lucene.index.IndexReader.open(IndexReader.java:316) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1084) ... 33 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) ... 48 more I can see that the error is going down to lucene and java, but I don't have a clue what should I do... Any suggestions? Thanks and merry christmas:) Rok
