Hi Bill, MMapDirectory uses the file system cache of your operating system, which has following consequences: In Linux, top & free should normally report only *few* free memory, because the O/S uses all memory not allocated by applications to cache disk I/O (and shows it as allocated, so having 0% free memory is just normal on Linux and also Windows). If you have other applications or Lucene/Solr itself that allocate lot's of heap space or malloc() a lot, then you are reducing free physical memory, so reducing fs cache. This depends also on your swappiness parameter (if swappiness is higher, inactive processes are swapped out easier, default is 60% on linux - freeing more space for FS cache - the backside is of course that maybe in-memory structures of Lucene and other applications get pages out).
You will only see no paging at all if all memory allocated all applications + all mmapped files fit into memory. But paging in/out the mmapped Lucene index is muuuuuch cheaper than using SimpleFSDirectory or NIOFSDirectory. If you use SimpleFS or NIO and your index is not in FS cache, it will also read it from physical disk again, so where is the difference. Paging is actually cheaper as no syscalls are involved. If you want as much as possible of your index in physical RAM, copy it to /dev/null regularily and buy more RUM :-) ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Bill Bell [mailto:billnb...@gmail.com] > Sent: Friday, July 20, 2012 5:17 AM > Subject: Re: How to setup SimpleFSDirectoryFactory > > Thanks. Are you saying that if we run low on memory, the MMapDirectory will > s=op using it? The least used memory will be removed from the OS > automaticall=? Isee some paging. Wouldn't paging slow down the querying? > > My index is 10gb and every 8 hours we get most of it in shared memory. The > m=mory is 99 percent used, and that does not leave any room for other apps. = > Other implications? > > Sent from my mobile device > 720-256-8076 > > On Jul 19, 2012, at 9:49 AM, "Uwe Schindler" <u...@thetaphi.de> wrote: > > > Read this, then you will see that MMapDirectory will use 0% of your Java > H=ap space or free system RAM: > > > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm > > l > > > > Uwe > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > >> -----Original Message----- > >> From: William Bell [mailto:billnb...@gmail.com] > >> Sent: Tuesday, July 17, 2012 6:05 AM > >> Subject: How to setup SimpleFSDirectoryFactory > >> > >> We all know that MMapDirectory is fastest. However we cannot always > >> use i= since you might run out of memory on large indexes right? > >> > >> Here is how I got iSimpleFSDirectoryFactory to work. Just set - > >> Dsolr.directoryFactory=solr.SimpleFSDirectoryFactory. > >> > >> Your solrconfig.xml: > >> > >> <directoryFactory name="DirectoryFactory" > >> class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/> > >> > >> You can check it with http://localhost:8983/solr/admin/stats.jsp > >> > >> Notice that the default for Windows 64bit is MMapDirectory. Else > >> NIOFSDirectory except for WIndows.... It would be nicer if we just > >> set it=all up with a helper in solrconfig.xml... > >> > >> if (Constants.WINDOWS) { > >> if (MMapDirectory.UNMAP_SUPPORTED && Constants.JRE_IS_64BIT) > >> return new MMapDirectory(path, lockFactory); > >> else > >> return new SimpleFSDirectory(path, lockFactory); > >> } else { > >> return new NIOFSDirectory(path, lockFactory); > >> } > >> } > >> > >> > >> > >> -- > >> Bill Bell > >> billnb...@gmail.com > >> cell 720-256-8076 > > > >
