I think my problem has something to do with it. But to be honest, I am not totally sure that is *the* *only* problem (with 10G of dump data, it is just hard to find useful things even with all kinds of tools). I manually patched the code and I will be able to put the code under stress test (the same level when OutOfMemory happens to me last time) next Monday and see how it behaves.
I will keep you posted. Thanks, Yong -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Emmanuel Lecharny Sent: Friday, May 15, 2009 9:33 AM To: [email protected] Subject: Re: memory leak? -- help needed FYI, this finalize method was added on revision 190270, three years and 11 months ago ... http://svn.apache.org/viewvc/directory/network/trunk/src/java/org/apache/mina/common/AbstractIoFilterChain.java?r1=190270&r2=190269&pathrev=190270 So it has been present in the code base before the first version was released... I would be very surprised if nobody has found this problem since then. There must but something else going wrong... I'm not saying that you are wrong, I don't like this method either, but I want to be sure that we are focusing on the real problem here. On Fri, May 15, 2009 at 6:21 PM, Emmanuel Lecharny <[email protected]>wrote: > Thanks for the detailed explaination. I will spend some time tonight to > build a sample application exhibiting the behaviour you are exposing (unless > you have already a portion of code for that !). > > In any case, the 'finalize' method is not one of my favorite method... Not > only it has a very specific usage, but there is no guarantee whatsoever to > be called by the GC. I will check if it's necessary to have it. > > > On Fri, May 15, 2009 at 6:14 PM, Yongxing Wang <[email protected]>wrote: > >> Emmanuel, >> >> >> When client close connection, session should be discarded and it is kind >> of so. The only thing, I think, is because that the "finalize" method on >> DefaultIoFilterChain. Because it has such method, JVM can't claim the memory >> back right away because inside of the finalize code, it is impossible the >> very same object could be resurrected (because we users provide the >> implementation), so GC has to treat them quite differently (possibly go over >> multiple GC cycles before fully discard it). That's why the chain is still >> referenced by the Finalizer and so the session. >> >> In theory, those objects should be eventually recycled. But the timing is >> totally un-deterministic and they can stay there for unnecessarily long. >> This is even true to Session objects. They are long live objects and most >> likely stay in PermGen, where GC recycles much less frequently. >> >> You can do a simple test. Use a client/server example in MINA code, start >> the server, start one client and let the client do some work and then close >> the client. Wait for 10 minutes, check the memory map of the server. Chances >> are, you can still find that Session object. If you trace it to "GC rootset >> exclude weak references", you will find that the session is referenced by >> the chain, which is referenced by Finalizer. >> >> Now remove the "finalize" method in the chain and do the same test, you >> will find a different result. >> >> Regardless of whether that is the true source of my memory leak, hopefully >> you can recognize the "finalize" method is evil. In the case of the filter >> chain, I don't see any reason why it is adopted. Getting rid of that would >> be a very good idea. >> >> Thanks, >> Yong >> >> Here is how my chain is initialized. >> ---------------------------------------------------------------------- >> >> DefaultIoFilterChainBuilder chain = acceptor.getFilterChain(); >> chain.addLast("codec", new ProtocolCodecFilter(new >> ScMessageProtocolCodecFactory())); >> >> chain.addLast("logger", new LoggingFilter()); >> chain.addLast("monitor", new ProtocolMonitorFilter()); >> >> ExecutorService threadPool = new OrderedThreadPoolExecutor(4); >> chain.addLast("executor", new ExecutorFilter(threadPool)); >> acceptor.setHandler(new CentralDispatchHandler()); >> >> >> -----Original Message----- >> From: Emmanuel Lecharny [mailto:[email protected]] On Behalf Of >> Emmanuel Lecharny >> Sent: Friday, May 15, 2009 1:17 AM >> To: [email protected] >> Subject: Re: memory leak? -- help needed >> >> Hi, >> >> this is very strange... If the client close the connection before the >> message is sent to him, then the session should be discarded. >> >> Could you provide the code which initialize the chain you are using ? >> >> Also could you run your test with the latest version (ie, the trunk)? We >> have fixed some bad issues lately about the way messages are handled. >> >> Thanks ! >> >> >> Yongxing Wang wrote: >> >> > In sessionClosed() method in DefaultIoFilterChain, it will do: >> > >> > s.getWriteRequestQueue().dispose(session); >> > >> > >> > There are only two classes that implements WriteRequestQueue, one is >> CloseAwareWriteQueue and another is DefaultWriteRequestQueue. >> CloseAwareWriteQueue is just a wrapper on top of DefaultWriteRequestQueue >> where the dispose() method does nothing. >> > >> > public void dispose(IoSession session) { >> > } >> > >> > >> > So I have a theory for the mem leak: >> > >> > 1: Client request something, >> > 2: Server fetch the data and call session.write(), the data got queued >> in the wirteRequestQueue. >> > 3: Before the data got actually flushed, the client drops the connection >> > 4: MINA fires sessionClosed() as expect, but the dispose method didn't >> actually clear the queue. >> > 5: Because of the session still got referenced by the Finalizer, the >> huge amount of data just hang around. >> > >> > >> > Am I right that the dispose() method should clear the queue? >> > >> > Thanks, >> > Yong >> > >> > >> > >> > >> > -----Original Message----- >> > From: Yongxing Wang >> > Sent: Thursday, May 14, 2009 9:31 AM >> > To: '[email protected]' >> > Subject: RE: memory leak? -- help needed >> > >> > Yes, I will definitely run it against one of those tools (my Jprofile >> evaluation expired, need to work it out). >> > >> > I guess last night my brain was fried and I didn't provide enough >> information. The dump analysis told me that there are still more than 7K >> NioSocketSession (referenced by java.lang.Finalizer) hanging around three >> hours after I the process out of LB. This probably related to one of my >> earlier post: >> > >> > "Looking at the source code, it is caused by DefaultIoFilterChain >> (because it has a finalize method)." -- the chain holds a reference to the >> session. >> > >> > I tried to trace the "DefaultWriteFuture" to the GC root. With 10G of >> dump data, the query never returns. >> > >> > I will continue to do more analysis. At the mean time, is there a reason >> that you have to use "finalize" method in the DefaultIoFilterChain? >> > >> > Yong >> > >> > >> > -----Original Message----- >> > From: Emmanuel Lecharny [mailto:[email protected]] On Behalf Of >> Emmanuel Lecharny >> > Sent: Wednesday, May 13, 2009 11:47 PM >> > To: [email protected] >> > Subject: Re: memory leak? -- help needed >> > >> > Yongxing Wang wrote: >> > >> >> Gurus, >> >> >> >> >> > hi, >> > >> > you should use a tool to determine if the memory leak you are >> > experiencing is part of your own code. You have plenty of them out there >> > : JProfiler, OptimizeIt, YourKit, ... >> > >> > This will give you some insight about the part of the code where this >> > memory leak is likely to be created. >> > >> > There is not much we can do with the provided information atm... >> > >> > >> >> >> -- >> -- >> cordialement, regards, >> Emmanuel Lécharny >> www.iktek.com >> directory.apache.org >> >> >> > > > -- > Regards, > Cordialement, > > Emmanuel Lécharny > www.iktek.com > -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
