Hi, For a full description of Lucene & MMap, see: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Quote: "How to configure my operating system and Java VM to make optimal use of MMapDirectory? First of all, default settings in Linux distributions and Solaris/Windows are perfectly fine. But there are some paranoid system administrators around, that want to control everything (with lack of understanding). Those limit the maximum amount of virtual address space that can be allocated by applications. So please check that “ulimit -v” and “ulimit -m” both report “unlimited”, otherwise it may happen that MMapDirectory reports “mmap failed” while opening your index. If this error still happens on systems with lot’s of very large indexes, each of those with many segments, you may need to tune your kernel parameters in /etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may need to raise it. I think, for Windows and Solaris systems there are similar settings available, but it is up to the reader to find out how to use them. For configuring your Java VM, you should rethink your memory requirements: Give only the really needed amount of heap space and leave as much as possible to the O/S. As a rule of thumb: Don’t use more than ¼ of your physical memory as heap space for Java running Lucene/Solr, keep the remaining memory free for the operating system cache. If you have more applications running on your server, adjust accordingly. As usual the more physical memory the better, but you don’t need as much physical memory as your index size. The kernel does a good job in paging in frequently used pages from your index. A good possibility to check that you have configured your system optimally is by looking at both "top" (and correctly interpreting it, see above) and the similar command "iotop" (can be installed, e.g., on Ubuntu Linux by "apt-get install iotop"). If your system does lots of swap in/swap out for the Lucene process, reduce heap size, you possibly used too much. If you see lot's of disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to be paged in/out all the time, and finally: buy SSDs." This also has the recommened settings for ulimit and other sysctls. Changing the chunk size does not help on 64 bit platforms, it should be 1 GiB there. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Harald Kirsch [mailto:harald.kir...@raytion.com] > Sent: Monday, August 18, 2014 1:19 PM > To: java-user@lucene.apache.org > Subject: Re: OutOfMemory when initializing MMapIndexInput on lucene > 3.6.2 > > ulimit -v unlimited > > might help, see > > http://stackoverflow.com/questions/8892143/error-when-opening-a- > lucene-index-map-failed > > Harald. > > On 18.08.2014 13:10, Shlomit Rosen wrote: > > Hi all, > > > > Using lucene 3.6.2, we are trying to search a pretty small collection. > > To open the directory we use Mmap since we are running on a 64 bit > > linux machine, and we usually get much better results than using > > SimpleFS or NIO. > > Although the collection is only a few GB in size, we are getting an > > OOM exception right from the start: > > > > IQQS0003E The searchable object for collection > > > /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448 > 3BD70 > > 0CA08571491C > > cannot be loaded. > > Causes of the problem: > > IQQS0004E The reader for index > > > /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448 > 3BD70 > > 0CA08571491C/data/text > > cannot be created. > > IQQG0020E java.io.IOException: Map failed > > IQQG0020E java.lang.OutOfMemoryError: Map failed > > sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784) > > > > > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MM > apDirectory.java:270) > > > > > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22 > 0 > > ) > > > > > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreRe > aders.java:96) > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) > > > > org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.j > > ava:105) > > > > > org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDir > ectoryReader.java:27) > > > > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java: > > 78) > > > > > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo > s.java:709) > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72) > > org.apache.lucene.index.IndexReader.open(IndexReader.java:375) > > ... > > > > Collection details: > > > > File Name File Size > > --------- --------- > > _9d.prx 700520240 bytes (668.07 MB) > > _6a.prx 688987818 bytes (657.07 MB) > > _im.prx 678185165 bytes (646.77 MB) > > _37.prx 672422515 bytes (641.27 MB) > > _rv.prx 672034961 bytes (640.90 MB) > > _fj.prx 669692961 bytes (638.67 MB) > > _cg.prx 666485509 bytes (635.61 MB) > > _os.prx 661239341 bytes (630.61 MB) > > _lp.prx 656960684 bytes (626.53 MB) > > _9d.frq 237580704 bytes (226.57 MB) > > _6a.frq 233601024 bytes (222.78 MB) > > _im.frq 229835093 bytes (219.19 MB) > > _37.frq 227920880 bytes (217.36 MB) > > _rv.frq 227857650 bytes (217.30 MB) > > _fj.frq 227030974 bytes (216.51 MB) > > _cg.frq 225955651 bytes (215.49 MB) > > _os.frq 224161119 bytes (213.78 MB) > > _lp.frq 222748045 bytes (212.43 MB) > > _tl.cfs 104218657 bytes (99.39 MB) > > _s2.cfs 99241207 bytes (94.64 MB) > > _ta.cfs 96972944 bytes (92.48 MB) > > _sz.cfs 95444312 bytes (91.02 MB) > > _sd.cfs 95334907 bytes (90.92 MB) > > _so.cfs 94020302 bytes (89.66 MB) > > _9d.fdt 67804161 bytes (64.66 MB) > > _6a.fdt 66670600 bytes (63.58 MB) > > _im.fdt 65635643 bytes (62.60 MB) > > _37.fdt 65082068 bytes (62.07 MB) > > _rv.fdt 65027960 bytes (62.02 MB) > > _fj.fdt 64820141 bytes (61.82 MB) > > _cg.fdt 64519880 bytes (61.53 MB) > > _os.fdt 63995985 bytes (61.03 MB) > > _lp.fdt 63578365 bytes (60.63 MB) > > _ts.cfs 16675689 bytes (15.90 MB) > > _tq.cfs 16630852 bytes (15.86 MB) > > _tn.cfs 10548412 bytes (10.06 MB) > > _to.cfs 9935882 bytes (9.48 MB) > > _tp.cfs 9667762 bytes (9.22 MB) > > _tm.cfs 9473559 bytes (9.03 MB) > > _tk.cfs 8742824 bytes (8.34 MB) > > _9d.tis 8027312 bytes (7.66 MB) > > _6a.tis 7898747 bytes (7.53 MB) > > _im.tis 7775053 bytes (7.41 MB) > > _37.tis 7714470 bytes (7.36 MB) > > _rv.tis 7702330 bytes (7.35 MB) > > _fj.tis 7682979 bytes (7.33 MB) > > _cg.tis 7649182 bytes (7.29 MB) > > _os.tis 7585214 bytes (7.23 MB) > > _lp.tis 7537160 bytes (7.19 MB) > > _tt.cfs 1832496 bytes (1.75 MB) > > _9d.fdx 858540 bytes (838.42 KB) > > _6a.fdx 844204 bytes (824.42 KB) > > _im.fdx 831084 bytes (811.61 KB) > > _37.fdx 824068 bytes (804.75 KB) > > _rv.fdx 823380 bytes (804.08 KB) > > _fj.fdx 820756 bytes (801.52 KB) > > _cg.fdx 816972 bytes (797.82 KB) > > _os.fdx 810332 bytes (791.34 KB) > > _lp.fdx 805044 bytes (786.18 KB) > > _tr.cfs 640731 bytes (625.71 KB) > > _tu.cfs 419728 bytes (409.89 KB) > > _9d.nrm 214638 bytes (209.61 KB) > > _6a.nrm 211054 bytes (206.11 KB) > > _im.nrm 207774 bytes (202.90 KB) > > _37.nrm 206020 bytes (201.19 KB) > > _rv.nrm 205848 bytes (201.02 KB) > > _fj.nrm 205192 bytes (200.38 KB) > > _cg.nrm 204246 bytes (199.46 KB) > > _os.nrm 202586 bytes (197.84 KB) > > _lp.nrm 201264 bytes (196.55 KB) > > _9d.tii 84483 bytes (82.50 KB) > > _6a.tii 83166 bytes (81.22 KB) > > _im.tii 81863 bytes (79.94 KB) > > _37.tii 81277 bytes (79.37 KB) > > _rv.tii 81113 bytes (79.21 KB) > > _fj.tii 80939 bytes (79.04 KB) > > _cg.tii 80709 bytes (78.82 KB) > > _os.tii 79774 bytes (77.90 KB) > > _lp.tii 79369 bytes (77.51 KB) > > segments_qj 5977 bytes (5.84 KB) > > _37.fnm 86 bytes > > _6a.fnm 86 bytes > > _9d.fnm 86 bytes > > _cg.fnm 86 bytes > > _fj.fnm 86 bytes > > _im.fnm 86 bytes > > _lp.fnm 86 bytes > > _os.fnm 86 bytes > > _rv.fnm 86 bytes > > > > > > > > Does this seem like a reasonable behavior or is this a bug? > > It seems to me that with this small segments, memory should not be an > > issue...? > > > > > > We haven't seen this exception in previous testing. > > In the lucene documentation I found that tweaking the chunk size might > > help, But this code goes into customer production environment, and > > this option is not feasible. > > Is there anything else I can check ahead of time (except for if this > > is a > > 32/64 bit platform) to help me decide > > Whether or not Mmap should be used? > > > > Thanks in advance, > > Shlomit > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
