It is hopeless to talk to both of you, you don't understand virtual memory:
> I get a similar situation using Windows 2008 and Solr 3.6. Memory using > mmap=is never released. Even if I turn off traffic and commit and do a manual > gc= If the size of the index is 3gb then memory used will be heap + 3gb of > sha=ed used. If I use a 6gb index I get heap + 6gb. That is expected, but we are talking not about allocated physical memory, we are talking about allocated ADDRESS SPACE and you have 2^47 of that on 64bit platforms. There is no physical memory wasted or allocated - please read the blog post a third, forth, fifth... or tenth time, until it is obvious. You should also go back to school and take a course on system programming and operating system kernels. Every CS student gets that taught in his first year (at least in Germany). Java's GC has nothing to do with that - as long as the index is open, ADDRESS SPACE is assigned. We are talking not about memory nor Java heap space. > If I turn off > MMapDirectory=actory it goes back down. When is the MMap supposed to > release memory ? It o=ly does it on JVM restart now. Can you please stop spreading nonsense about MMapDirectory with no knowledge behind? http://www.linuxatemyram.com/ - Also applies to Windows. Uwe > Bill Bell > Sent from mobile > > > On Jul 22, 2012, at 6:21 AM, geetha anjali <anjaliprabh...@gmail.com> > wrote:= > > It happens in 3.6, for this reasons I thought of moving to solandra. > > If I do a commit, the all documents are persisted with out any issues. > > There is no issues in terms of any functionality, but only this > > happens i= increase in physical RAM, goes higher and higher and stop > > at maximum and i= never comes down. > > > > Thanks > > > > On Sun, Jul 22, 2012 at 3:38 AM, Lance Norskog <goks...@gmail.com> > wrote: > > > >> Interesting. Which version of Solr is this? What happens if you do a > >> commit? > >> > >> On Sat, Jul 21, 2012 at 8:01 AM, geetha anjali > <anjaliprabh...@gmail.com>=>> wrote: > >>> Hi uwe, > >>> Great to know. We have files indexing 10000/min. After 30 mins I see > >>> all=>>> my physical memory say its 100 percentage used(windows). On > >>> deep investigation found that mmap is not releasing os files handles. Do > you find this behaviour? > >>> > >>> Thanks > >>> > >>> On 20 Jul 2012 14:04, "Uwe Schindler" <u...@thetaphi.de> wrote: > >>> > >>> Hi Bill, > >>> > >>> MMapDirectory uses the file system cache of your operating system, > >>> which=>> has following consequences: In Linux, top & free should > >>> normally report only=>>> *few* free memory, because the O/S uses all > >>> memory not allocated by applications to cache disk I/O (and shows it > >>> as allocated, so having 0% > >> free > >>> memory is just normal on Linux and also Windows). If you have other > >>> applications or Lucene/Solr itself that allocate lot's of heap space > >>> or > >>> malloc() a lot, then you are reducing free physical memory, so > >>> reducing > >> fs > >>> cache. This depends also on your swappiness parameter (if swappiness > >>> is higher, inactive processes are swapped out easier, default is 60% > >>> on > >> linux - > >>> freeing more space for FS cache - the backside is of course that > >>> maybe in-memory structures of Lucene and other applications get pages > out). > >>> > >>> You will only see no paging at all if all memory allocated all > >> applications > >>> + all mmapped files fit into memory. But paging in/out the mmapped > >>> + Lucen= > >>> index is muuuuuch cheaper than using SimpleFSDirectory or > >> NIOFSDirectory. If > >>> you use SimpleFS or NIO and your index is not in FS cache, it will > >>> also > >> read > >>> it from physical disk again, so where is the difference. Paging is > >> actually > >>> cheaper as no syscalls are involved. > >>> > >>> If you want as much as possible of your index in physical RAM, copy > >>> it t= /dev/null regularily and buy more RUM :-) > >>> > >>> > >>> ----- > >>> Uwe Schindler > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >>> eMail: uwe@thetaphi... > >>> > >>>> From: Bill Bell [mailto:billnb...@gmail.com] > >>>> Sent: Friday, July 20, 2012 5:17 AM > >>>> Subject: Re: ... > >>>> s=op using it? The least used memory will be removed from the OS > >>>> automaticall=? Isee some paging. Wouldn't paging slow down the > queryi=g? > >>> > >>>> > >>>> My index is 10gb and every 8 hours we get most of it in shared memory. > >> The > >>>> m=mory is 99 percent used, and that does not leave any room for > >>>> other=>>> apps. = > >>> > >>>> Other implications? > >>>> > >>>> Sent from my mobile device > >>>> 720-256-8076 > >>>> > >>>> On Jul 19, 2012, at 9:49 A... > >>>> H=ap space or free system RAM: > >>> > >>>>> > >>>>> > >> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.ht > >> m > >>>>> l > >>>>> > >>>>> Uwe > >>>>> ... > >>>>>> use i= since you might run out of memory on large indexes right? > >>> > >>>>>> > >>>>>> Here is how I got iSimpleFSDirectoryFactory to work. Just set - > >>>>>> Dsolr.directoryFactor... > >>>>>> set it=all up with a helper in solrconfig.xml... > >>> > >>>>>> > >>>>>> if (Constants.WINDOWS) { > >>>>>> if (MMapDirectory.UNMAP_SUPPORTED && Constants.JRE_IS_64... > >> > >> > >> > >> -- > >> Lance Norskog > >> goks...@gmail.com > >>
