Without looking at the code heavily, I think the problem here involves wrapped readers like slowMultiReaderWrapper.
Take a look at this code in branch-3x. you will notice that we *never* both wrap with Slow and add an executor service. But the real problem is probably that newSearcher should not wrap readers at all. instead I suppose tests should explicitly separately call reader = maybeWrapReader(reader) or something like that, so that they actually close() the wrapped reader, (not just its inner children). This will ensure the close hook is actually called... again the issue is that the test never closes the slow-wrapped reader, because its not even aware of its existence. On Fri, Mar 2, 2012 at 5:22 AM, Dawid Weiss <[email protected]> wrote: > This leak is really frequent if you preallocate threads on the > execution service. This fragment: > > r.addReaderClosedListener(new ReaderClosedListener() { > @Override > public void onClose(IndexReader reader) { > System.out.println("S:" + ex); > shutdownExecutorService(ex); > } > }); > > subscribes to the on-close event. I don't see how these listeners can > be properly called in case of something like > SlowCompositeReaderWrapper -- it accepts listeners and subscribes them > to itself but never does anything in doClose. The listener should be > subscribed to the delegate (I think?) but then the method to add > listeners is final up the class hierarchy. > > ? > > Dawid > > > On Thu, Mar 1, 2012 at 12:20 PM, Uwe Schindler <[email protected]> wrote: >> Hi, >> >> Yeah it's strange. I checked the code: It either does wrap and runs >> single-thread searches or it does *not* wrap and runs several threads. So >> theoretically it should work correctly... We have to check, if all >> IndexReaders are correctly closed and the listeners are called. >> >> Uwe >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: [email protected] >> >> >>> -----Original Message----- >>> From: [email protected] [mailto:[email protected]] On Behalf Of >>> Dawid Weiss >>> Sent: Thursday, March 01, 2012 12:02 PM >>> To: [email protected] >>> Subject: Re: ThreadPool threads leaking to suite scope. >>> >>> I don't know how to fix it, Uwe, but I know there's definitely something >>> not all >>> right with it because threads just keep accumulating (as new searchers are >>> created). >>> >>> I've pushed a static seed for which this is repeatable; this is a heavily >>> worked-on >>> branch but it may lead you to how to fix this: >>> >>> git clone git://github.com/dweiss/lucene_solr.git >>> git checkout 935e1e9e9a350d6b35b23c4545caf78e82b42747 >>> >>> try to run TestPhraseQuery (you'll need -ea in Eclipse). >>> >>> Dawid >>> >>> On Thu, Mar 1, 2012 at 11:55 AM, Uwe Schindler <[email protected]> wrote: >>> > I think the problem in newSearcher ist hat sometimes the reader is >>> > wrapped. >>> If its wrapped, the underlying reader is only closed, not the wrapper. But >>> the >>> listener is added to the wrapper. We should add the listener to the original >>> inner reader. >>> > >>> > Uwe >>> > >>> > ----- >>> > Uwe Schindler >>> > H.-H.-Meier-Allee 63, D-28213 Bremen >>> > http://www.thetaphi.de >>> > eMail: [email protected] >>> > >>> > >>> >> -----Original Message----- >>> >> From: Dawid Weiss [mailto:[email protected]] >>> >> Sent: Thursday, March 01, 2012 11:51 AM >>> >> To: [email protected] >>> >> Subject: Re: ThreadPool threads leaking to suite scope. >>> >> >>> >> > 1) initialize threads eagerly; use ThreadPoolExecutor and call >>> >> > prestartAllCoreThreads. this could be applied to LTC on the trunk. >>> >> >>> >> I did this but threads still leak out from unclosed readers created >>> >> by LTC#newSearcher. I don't know why, but this isn't called -- >>> >> >>> >> r.addReaderClosedListener(new ReaderClosedListener() { >>> >> @Override >>> >> public void onClose(IndexReader reader) { >>> >> shutdownExecutorService(ex); >>> >> } >>> >> }); >>> >> >>> >> Clues? >>> >> >>> >> Dawid >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: [email protected] For >>> >> additional commands, e-mail: [email protected] >>> > >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: [email protected] For >>> > additional commands, e-mail: [email protected] >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] For additional >>> commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > -- lucidimagination.com --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
