So my blog post, last section, helped? I think the ulimits came from there. What distribution do you use that ulimit was actually limited - or was it some sysadmin doing this? :-)
We should maybe refer to this blog post from docs or create a copy of the page inside lucene's distribution! Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > Sent: Wednesday, August 15, 2012 2:07 PM > To: java-user@lucene.apache.org > Subject: Re: howto run CheckIndex on huge index size > > Hi Uwe, > > index size is: > -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt > -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx > -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm > -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq > -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm > -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx > -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii > -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis > -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u > -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen > > Setting ulimit -v and ulimit -m to unlimited solved this and I can run CheckIndex > without any java options. > > The setting with ulimit should be pinned on lucene and solr pages! > > Thanks for help, > Bernd > > > Am 15.08.2012 13:34, schrieb Uwe Schindler: > > You don't get a heap-related OOM in your stack trace, it is "Map > > failed" - caused by MMapDirectory. You don't have enough virtual > > memory to map the index into address space. I think your heap is way > > too mch (-Xmx25g is way too big for any existing index and drives GC crazy). > How big is your index? > > > > See > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm > > l > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > >> -----Original Message----- > >> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > >> Sent: Wednesday, August 15, 2012 1:25 PM > >> To: java-user@lucene.apache.org > >> Subject: howto run CheckIndex on huge index size > >> > >> > >> I'm trying to run CheckIndex as seperate tool on a large index to get > >> nice > > infos > >> about number of terms, number of tokens, ... but always get OOM > exception. > >> > >> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g > >> > >> > >> Any idea how to use CheckIndex on huge index size? > >> > >> > >> > >> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index > >> Segments file=segments_5u numSegments=1 version=3.6.1 > >> format=FORMAT_3_1 [Lucene 3.1+] > >> 1 of 1: name=_2rhe docCount=37821132 > >> compound=false > >> hasProx=true > >> numFiles=8 > >> size (MB)=182.992,35 > >> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, > >> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 > >> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1, > >> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.} > >> no deletions > >> test: open reader.........FAILED > >> WARNING: fixIndex() would remove reference to this segment; full > >> exception: > >> java.io.IOException: Map failed > >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) > >> at > >> > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec > >> t > >> ory.java:271) > >> at > >> > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) > >> at > >> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104) > >> at > >> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders. > >> jav > >> a:83) > >> at > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > >> at > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) > >> at > > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523) > >> at > >> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064) > >> Caused by: java.lang.OutOfMemoryError: Map failed > >> at sun.nio.ch.FileChannelImpl.map0(Native Method) > >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745) > >> ... 8 more > >> WARNING: 1 broken segments (containing 37821132 documents) detected > >> WARNING: would write new segments file, and 37821132 documents would > >> be lost, if -fix were specified > >> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > -- > ************************************************************* > Bernd Fehling Universitätsbibliothek Bielefeld > Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie > Universitätsstr. 25 und Wissensmanagement > 33615 Bielefeld > Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de > > BASE - Bielefeld Academic Search Engine - www.base-search.net > ************************************************************* > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
