The answer is simple: It load the whole index into RAM because it has the RAM available for free use. If there would be apps running really using that RAM, this would not happen.
There is no difference if you use another Lucene directory implementation, merging is a heavy IO operation, so it will always load all of the index into the FS cache while merging (if there is space in it). You will see the same behavior, if you do "dd if=largeindexfile of=/dev/null". After that it will also consume all of filesystem cache because it transferred the whole file through RAM. The only difference here is, that there is no *address space* allocated. But as stated in the article, address space is for free on 64 bit OS. In no case, resident RAM will be raised by this. Also heap space stays the same. Teach your system administrator. And it is also not a windows problem. I have a 20 Gigabytes index on my windows 7 64 bit machine and it is still useable and doesn’t swap while merging (although I have other apps running).. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: v.se...@lombardodier.com [mailto:v.se...@lombardodier.com] > Sent: Thursday, January 17, 2013 5:12 PM > To: 'java-user@lucene.apache.org' > Cc: s.seym...@lombardodier.com; y.al...@lombardodier.com; > c.di...@lombardodier.com > Subject: mmap loads the entire index into memory during > forceMergeDeletes/forceMerge(int) > > Hi, > On a 256 Gb RAM machine, we have half of our IT system running. > Part of it, are 2 lucene applications, managing each a an approximate 100 Gb > index. > These applications are used to index logging events, and every night there is > a purge, followed by a forceMergeDeletes to reclaim disk space (and I have > configured forceMergeDeletesPctAllowed=0 to make sure I really do get the > space back). > We have had production incidents recently, because the resident size would > go as high as 100 Gb on each lucene app process during forceMergeDeletes. > After an investigation I figured out the default behavior of > FSDirectory.open(File), instantiating a MMapDirectory on 64 bits windows. > I read http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on- > 64bit.html, and I guess I am seeing something different than what is stated in > the article: "MMapDirectory will not load the whole index into physical > memory. Why should it do this? We just ask the operating system to map the > file into address space for easy access, by no means we are requesting more. > Java and the O/S optionally provide the option to try loading the whole file > into RAM (if enough is available), but Lucene does not use that option (we > may add this possibility in a later version)." > > At least, in my case it does load the entire index into memory. > > This behavior caught everybody by surprise, even app servers administrators > that are used to monitor heaps on java machines, not resident memory. > Mmap is appropriate when the lucene app is running alone on a physical box > or a vm, because if it wants to load the entire index into memory, it will > swap > out its own pages. But when you use a shared machine, then you start > hurting the other app servers. > > I think you should raise awareness on mmap usage when dealing with a large > index. > > Regards, > > vince > > > ************************ DISCLAIMER ************************ > This message is intended only for use by the person to whom it is addressed. > It may contain information that is privileged and confidential. Its content > does not constitute a formal commitment by Lombard Odier & Cie or any of > its branches or affiliates. If you are not the intended recipient of this > message, kindly notify the sender immediately and destroy this message. > Thank You. > ********************************************************** > ******* --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
