Hi, > Thank you very much for the quick reply :) > > Just got a word back from our consumer that setting ulimit -v and ulimit -m to > unlimited solved there problem.
Great! > They did raise a question about the best practice for this. > Assuming a customer will not want to set these flags to unlimit, but just > increase their value to the minimum required.... > Is there a rule of thumb as to how to calculate the limit size? > Would it be good enough to set it to the largest segment size of the largest > collection for example, or something of that effect? The absolute minimum required for -v would be a maximum virtual address space of the sum of the size of all indexes that are open at the same time (if they were static). If you do adds/deletes, you need 2 times of the size of all indexes. And finally, if you do horrible optimize/forceMerge and have indexes open for read at the same time, you may need 3 times. In addition the virtual address space gots fragmented over time on dynamic indexes, so not even that is enough. In general, -v should be unlimited in general! The setting only exists for older 32 bit platforms, because one process could easily map a large file and consume all available address space, which is in fact very limited to little more than 2 GiB (because the CPU has no larger pointers). On 64 bit CPUs, address space is almost free (depends on operating system and processor, in most cases you can use up to around 44 bits, see blog entry), so the limit makes no real sense at all. It might useful if you have multiple Lucene/Solr/Elasticsearch instances running on the same machine, each having indexes like 1 Terabyte of size and you want to prevent that one index accidently reserves all address space the other index might need to map its files, too.... -m can be bounded, it has not so much to do with virtual memory, it is about the resident memory. -m should at least have 1.5 times the heap size of the process (this is what Java itself allocates) plus some additional memory needed for read/write buffers, system libraries,... I just mentioned -m here and asked to set it to unlimited, to first try to prevent any problem that may arise. If you really need to set some value for -v (I don't recommend) or -m, a good approach would be to monitor the process with top and set something like VIRT as -v limit and RES as -m limit (with some buffer, like 1/2 of the common value). In general, limiting -v on 64 bit platforms is a very bad idea and counteracts the idea behind virtual address space. Uwe > Hi, > > For a full description of Lucene & MMap, see: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Quote: "How to configure my operating system and Java VM to make optimal > use of MMapDirectory? > First of all, default settings in Linux distributions and Solaris/Windows are > perfectly fine. But there are some paranoid system administrators around, > that want to control everything (with lack of understanding). > Those limit the maximum amount of virtual address space that can be > allocated by applications. So please check that ?ulimit -v? and ?ulimit -m? > both report ?unlimited?, otherwise it may happen that MMapDirectory > reports ?mmap failed? while opening your index. If this error still happens on > systems with lot?s of very large indexes, each of those with many segments, > you may need to tune your kernel parameters in > /etc/sysctl.conf: The default value of vm.max_map_count is 65530, you may > need to raise it. I think, for Windows and Solaris systems there are similar > settings available, but it is up to the reader to find out how to use them. > > For configuring your Java VM, you should rethink your memory > requirements: > Give only the really needed amount of heap space and leave as much as > possible to the O/S. As a rule of thumb: Don?t use more than ΒΌ of your > physical memory as heap space for Java running Lucene/Solr, keep the > remaining memory free for the operating system cache. If you have more > applications running on your server, adjust accordingly. As usual the more > physical memory the better, but you don?t need as much physical memory > as your index size. The kernel does a good job in paging in frequently used > pages from your index. > > A good possibility to check that you have configured your system optimally is > by looking at both "top" (and correctly interpreting it, see above) and the > similar command "iotop" (can be installed, e.g., on Ubuntu Linux by "apt-get > install iotop"). If your system does lots of swap in/swap out for the Lucene > process, reduce heap size, you possibly used too much. If you see lot's of > disk I/O, buy more RUM (Simon Willnauer) so mmapped files don't need to > be paged in/out all the time, and finally: buy SSDs." > > This also has the recommened settings for ulimit and other sysctls. > Changing the chunk size does not help on 64 bit platforms, it should be 1 GiB > there. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -----Original Message----- > > From: Harald Kirsch [mailto:harald.kir...@raytion.com] > > Sent: Monday, August 18, 2014 1:19 PM > > To: java-user@lucene.apache.org > > Subject: Re: OutOfMemory when initializing MMapIndexInput on lucene > > 3.6.2 > > > > ulimit -v unlimited > > > > might help, see > > > > http://stackoverflow.com/questions/8892143/error-when-opening-a- > > lucene-index-map-failed > > > > Harald. > > > > On 18.08.2014 13:10, Shlomit Rosen wrote: > > > Hi all, > > > > > > Using lucene 3.6.2, we are trying to search a pretty small collection. > > > To open the directory we use Mmap since we are running on a 64 bit > > > linux machine, and we usually get much better results than using > > > SimpleFS or NIO. > > > Although the collection is only a few GB in size, we are getting an > > > OOM exception right from the start: > > > > > > IQQS0003E The searchable object for collection > > > > > > /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448 > > 3BD70 > > > 0CA08571491C > > > cannot be loaded. > > > Causes of the problem: > > > IQQS0004E The reader for index > > > > > > /IndexStore/IndexArea/OS2_Document_20140813023606_9970347625CD448 > > 3BD70 > > > 0CA08571491C/data/text > > > cannot be created. > > > IQQG0020E java.io.IOException: Map failed > > > IQQG0020E java.lang.OutOfMemoryError: Map failed > > > sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:784) > > > > > > > > > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MM > > apDirectory.java:270) > > > > > > > > > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:22 > > 0 > > > ) > > > > > > > > > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreRe > > aders.java:96) > > > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) > > > > > > org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader > > > .j > > > ava:105) > > > > > > > > org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyD > > ir > > ectoryReader.java:27) > > > > > > > org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java: > > > 78) > > > > > > > > > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfo > > s.java:709) > > > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72) > > > > > > org.apache.lucene.index.IndexReader.open(IndexReader.java:375) > > > ... > > > > > > Collection details: > > > > > > File Name File Size > > > --------- --------- > > > _9d.prx 700520240 bytes (668.07 MB) > > > _6a.prx 688987818 bytes (657.07 MB) > > > _im.prx 678185165 bytes (646.77 MB) > > > _37.prx 672422515 bytes (641.27 MB) > > > _rv.prx 672034961 bytes (640.90 MB) > > > _fj.prx 669692961 bytes (638.67 MB) > > > _cg.prx 666485509 bytes (635.61 MB) > > > _os.prx 661239341 bytes (630.61 MB) > > > _lp.prx 656960684 bytes (626.53 MB) > > > _9d.frq 237580704 bytes (226.57 MB) > > > _6a.frq 233601024 bytes (222.78 MB) > > > _im.frq 229835093 bytes (219.19 MB) > > > _37.frq 227920880 bytes (217.36 MB) > > > _rv.frq 227857650 bytes (217.30 MB) > > > _fj.frq 227030974 bytes (216.51 MB) > > > _cg.frq 225955651 bytes (215.49 MB) > > > _os.frq 224161119 bytes (213.78 MB) > > > _lp.frq 222748045 bytes (212.43 MB) > > > _tl.cfs 104218657 bytes (99.39 MB) > > > _s2.cfs 99241207 bytes (94.64 MB) > > > _ta.cfs 96972944 bytes (92.48 MB) > > > _sz.cfs 95444312 bytes (91.02 MB) > > > _sd.cfs 95334907 bytes (90.92 MB) > > > _so.cfs 94020302 bytes (89.66 MB) > > > _9d.fdt 67804161 bytes (64.66 MB) > > > _6a.fdt 66670600 bytes (63.58 MB) > > > _im.fdt 65635643 bytes (62.60 MB) > > > _37.fdt 65082068 bytes (62.07 MB) > > > _rv.fdt 65027960 bytes (62.02 MB) > > > _fj.fdt 64820141 bytes (61.82 MB) > > > _cg.fdt 64519880 bytes (61.53 MB) > > > _os.fdt 63995985 bytes (61.03 MB) > > > _lp.fdt 63578365 bytes (60.63 MB) > > > _ts.cfs 16675689 bytes (15.90 MB) > > > _tq.cfs 16630852 bytes (15.86 MB) > > > _tn.cfs 10548412 bytes (10.06 MB) > > > _to.cfs 9935882 bytes (9.48 MB) > > > _tp.cfs 9667762 bytes (9.22 MB) > > > _tm.cfs 9473559 bytes (9.03 MB) > > > _tk.cfs 8742824 bytes (8.34 MB) > > > _9d.tis 8027312 bytes (7.66 MB) > > > _6a.tis 7898747 bytes (7.53 MB) > > > _im.tis 7775053 bytes (7.41 MB) > > > _37.tis 7714470 bytes (7.36 MB) > > > _rv.tis 7702330 bytes (7.35 MB) > > > _fj.tis 7682979 bytes (7.33 MB) > > > _cg.tis 7649182 bytes (7.29 MB) > > > _os.tis 7585214 bytes (7.23 MB) > > > _lp.tis 7537160 bytes (7.19 MB) > > > _tt.cfs 1832496 bytes (1.75 MB) > > > _9d.fdx 858540 bytes (838.42 KB) > > > _6a.fdx 844204 bytes (824.42 KB) > > > _im.fdx 831084 bytes (811.61 KB) > > > _37.fdx 824068 bytes (804.75 KB) > > > _rv.fdx 823380 bytes (804.08 KB) > > > _fj.fdx 820756 bytes (801.52 KB) > > > _cg.fdx 816972 bytes (797.82 KB) > > > _os.fdx 810332 bytes (791.34 KB) > > > _lp.fdx 805044 bytes (786.18 KB) > > > _tr.cfs 640731 bytes (625.71 KB) > > > _tu.cfs 419728 bytes (409.89 KB) > > > _9d.nrm 214638 bytes (209.61 KB) > > > _6a.nrm 211054 bytes (206.11 KB) > > > _im.nrm 207774 bytes (202.90 KB) > > > _37.nrm 206020 bytes (201.19 KB) > > > _rv.nrm 205848 bytes (201.02 KB) > > > _fj.nrm 205192 bytes (200.38 KB) > > > _cg.nrm 204246 bytes (199.46 KB) > > > _os.nrm 202586 bytes (197.84 KB) > > > _lp.nrm 201264 bytes (196.55 KB) > > > _9d.tii 84483 bytes (82.50 KB) > > > _6a.tii 83166 bytes (81.22 KB) > > > _im.tii 81863 bytes (79.94 KB) > > > _37.tii 81277 bytes (79.37 KB) > > > _rv.tii 81113 bytes (79.21 KB) > > > _fj.tii 80939 bytes (79.04 KB) > > > _cg.tii 80709 bytes (78.82 KB) > > > _os.tii 79774 bytes (77.90 KB) > > > _lp.tii 79369 bytes (77.51 KB) > > > segments_qj 5977 bytes (5.84 KB) > > > _37.fnm 86 bytes > > > _6a.fnm 86 bytes > > > _9d.fnm 86 bytes > > > _cg.fnm 86 bytes > > > _fj.fnm 86 bytes > > > _im.fnm 86 bytes > > > _lp.fnm 86 bytes > > > _os.fnm 86 bytes > > > _rv.fnm 86 bytes > > > > > > > > > > > > Does this seem like a reasonable behavior or is this a bug? > > > It seems to me that with this small segments, memory should not be > > > an issue...? > > > > > > > > > We haven't seen this exception in previous testing. > > > In the lucene documentation I found that tweaking the chunk size > > > might help, But this code goes into customer production environment, > > > and this option is not feasible. > > > Is there anything else I can check ahead of time (except for if this > > > is a > > > 32/64 bit platform) to help me decide Whether or not Mmap should be > > > used? > > > > > > Thanks in advance, > > > Shlomit > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
