Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-04 Thread Michael McCandless
Thanks for bringing closure Alon. In 3.0.3 each thread had a private terms dict cache, so that explains the high per-thread RAM usage. This was fixed at some 3.x release ... so we won't be fixing 3.0.3 at this point. Mike McCandless http://blog.mikemccandless.com On Fri, Jan 4, 2013 at 8:29 AM

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-04 Thread Alon Muchnick
Robert you were absolutely right , we had a problem with our deployment script and the Lucune Jars were not updated to 3.6.2 , they remained 3.0.3 ... after deploying the application again with the updated jars , the memory leak is gone ! (although i do see a lot more gc activity on the YoungGen

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-03 Thread Robert Muir
On Thu, Jan 3, 2013 at 12:16 PM, Alon Muchnick wrote: > value org.apache.lucene.index.TermInfosReader$ThreadResources ---> > > termInfoCache |org.apache.lucene.util.cache.SimpleLRUCache > termEnum |org.apache.lucene.index.SegmentTermEnum > You aren't using lucene 3.6.2 if you have th

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-03 Thread Michael McCandless
On Thu, Jan 3, 2013 at 12:16 PM, Alon Muchnick wrote: > hi Mike , > > at the peak there are 500 live threads going through Lucune (not all of > them at the same time , tomcat thread pool uses round robin ) ,regarding > the Directory impl we are using RAMDirectory. > the object that takes most of

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-03 Thread Alon Muchnick
hi Mike , at the peak there are 500 live threads going through Lucune (not all of them at the same time , tomcat thread pool uses round robin ) ,regarding the Directory impl we are using RAMDirectory. the object that takes most of the heap is the "hardRefs" WeakHashMap class member in the Close

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-02 Thread Michael McCandless
Hi Alon, CloseableThreadLocal has logic that tries to purge dead threads periodically (the countUntilPurge that you described) ... but it's only "best effort" / heuristic, and, indeed, if lots of threads have come through, and then suddenly all is quiet and the threads die off, then CTL will still

potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-02 Thread Alon Muchnick
hi MIke, thank you very much for replying . i am not sure when the doClose() method of the SegementReader will be called , but i think you are correct regarding the SegmentReader still being live , we have an singleton object with a hash map of several RamDirectory indexes and their IndexReader,

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-02 Thread Michael McCandless
If CloseableThreadLocal.hardRefs is still a live reference then it means the SegmentReader hasn't been closed yet, ie I think the chain you show below is from a still-open SegmentReader? Is it possible on upgrade to Tomcat 7 that more threads are doing searches (which will require more RAM since L

Re: potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-02 Thread Lance Norskog
There were memory leak problems with earlier versions of Java. You should upgrade to Java 6_30. Lance On 01/02/2013 05:26 AM, Alon Muchnick wrote: Hello All , we are using Lucune 3.6.2 in our web application on tomcat 5.5 and recently we started testing our application on tomcat 7, unfortunate

potential memory leak when using RAMDirectory ,CloseableThreadLocal and a thread pool .

2013-01-02 Thread Alon Muchnick
Hello All , we are using Lucune 3.6.2 in our web application on tomcat 5.5 and recently we started testing our application on tomcat 7, unfortunately we seem to encounter a memory link in Lucune's CloseableThreadLocal class , any help with solving the below issue would be much appreciated. we ar