Hi Uwe, index size is: -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
Setting ulimit -v and ulimit -m to unlimited solved this and I can run CheckIndex without any java options. The setting with ulimit should be pinned on lucene and solr pages! Thanks for help, Bernd Am 15.08.2012 13:34, schrieb Uwe Schindler: > You don't get a heap-related OOM in your stack trace, it is "Map failed" - > caused by MMapDirectory. You don't have enough virtual memory to map the > index into address space. I think your heap is way too mch (-Xmx25g is way > too big for any existing index and drives GC crazy). How big is your index? > > See http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] >> Sent: Wednesday, August 15, 2012 1:25 PM >> To: java-user@lucene.apache.org >> Subject: howto run CheckIndex on huge index size >> >> >> I'm trying to run CheckIndex as seperate tool on a large index to get nice > infos >> about number of terms, number of tokens, ... but always get OOM exception. >> >> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g >> >> >> Any idea how to use CheckIndex on huge index size? >> >> >> >> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index >> Segments file=segments_5u numSegments=1 version=3.6.1 >> format=FORMAT_3_1 [Lucene 3.1+] >> 1 of 1: name=_2rhe docCount=37821132 >> compound=false >> hasProx=true >> numFiles=8 >> size (MB)=182.992,35 >> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, os=Linux, >> lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 14:08:0 6, >> source=merge, os.arch=amd64, mergeMaxNumSegments=1, >> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.} >> no deletions >> test: open reader.........FAILED >> WARNING: fixIndex() would remove reference to this segment; full >> exception: >> java.io.IOException: Map failed >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) >> at >> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirect >> ory.java:271) >> at >> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) >> at >> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104) >> at >> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.jav >> a:83) >> at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) >> at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) >> at > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523) >> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064) >> Caused by: java.lang.OutOfMemoryError: Map failed >> at sun.nio.ch.FileChannelImpl.map0(Native Method) >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745) >> ... 8 more >> WARNING: 1 broken segments (containing 37821132 documents) detected >> WARNING: would write new segments file, and 37821132 documents would be >> lost, if -fix were specified >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > -- ************************************************************* Bernd Fehling Universitätsbibliothek Bielefeld Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie Universitätsstr. 25 und Wissensmanagement 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de BASE - Bielefeld Academic Search Engine - www.base-search.net ************************************************************* --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
