Sorry to get to you guys a little late on this issue, as it is not always easy to reproduce the problem.
During this extensive performance testing and close monitoring, one thing I can tell for sure is that under heavy load, the memory pattern looks so much nicer if we get rid of the "finalize" method. It contributes to the better overall system load. I strongly recommend getting rid of the "finalize" method in AbstractIoFilterChain.java. Thanks, Yong -----Original Message----- From: Yongxing Wang Sent: Friday, May 15, 2009 9:41 AM To: '[email protected]' Subject: RE: memory leak? -- help needed I think my problem has something to do with it. But to be honest, I am not totally sure that is *the* *only* problem (with 10G of dump data, it is just hard to find useful things even with all kinds of tools). I manually patched the code and I will be able to put the code under stress test (the same level when OutOfMemory happens to me last time) next Monday and see how it behaves. I will keep you posted. Thanks, Yong -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Emmanuel Lecharny Sent: Friday, May 15, 2009 9:33 AM To: [email protected] Subject: Re: memory leak? -- help needed FYI, this finalize method was added on revision 190270, three years and 11 months ago ... http://svn.apache.org/viewvc/directory/network/trunk/src/java/org/apache/mina/common/AbstractIoFilterChain.java?r1=190270&r2=190269&pathrev=190270 So it has been present in the code base before the first version was released... I would be very surprised if nobody has found this problem since then. There must but something else going wrong... I'm not saying that you are wrong, I don't like this method either, but I want to be sure that we are focusing on the real problem here. On Fri, May 15, 2009 at 6:21 PM, Emmanuel Lecharny <[email protected]>wrote: > Thanks for the detailed explaination. I will spend some time tonight to > build a sample application exhibiting the behaviour you are exposing (unless > you have already a portion of code for that !). > > In any case, the 'finalize' method is not one of my favorite method... Not > only it has a very specific usage, but there is no guarantee whatsoever to > be called by the GC. I will check if it's necessary to have it. > > > On Fri, May 15, 2009 at 6:14 PM, Yongxing Wang <[email protected]>wrote: > >> Emmanuel, >> >> >> When client close connection, session should be discarded and it is kind >> of so. The only thing, I think, is because that the "finalize" method on >> DefaultIoFilterChain. Because it has such method, JVM can't claim the memory >> back right away because inside of the finalize code, it is impossible the >> very same object could be resurrected (because we users provide the >> implementation), so GC has to treat them quite differently (possibly go over >> multiple GC cycles before fully discard it). That's why the chain is still >> referenced by the Finalizer and so the session. >> >> In theory, those objects should be eventually recycled. But the timing is >> totally un-deterministic and they can stay there for unnecessarily long. >> This is even true to Session objects. They are long live objects and most >> likely stay in PermGen, where GC recycles much less frequently. >> >> You can do a simple test. Use a client/server example in MINA code, start >> the server, start one client and let the client do some work and then close >> the client. Wait for 10 minutes, check the memory map of the server. Chances >> are, you can still find that Session object. If you trace it to "GC rootset >> exclude weak references", you will find that the session is referenced by >> the chain, which is referenced by Finalizer. >> >> Now remove the "finalize" method in the chain and do the same test, you >> will find a different result. >> >> Regardless of whether that is the true source of my memory leak, hopefully >> you can recognize the "finalize" method is evil. In the case of the filter >> chain, I don't see any reason why it is adopted. Getting rid of that would >> be a very good idea. >> >> Thanks, >> Yong >> >> Here is how my chain is initialized. >> ---------------------------------------------------------------------- >> >> DefaultIoFilterChainBuilder chain = acceptor.getFilterChain(); >> chain.addLast("codec", new ProtocolCodecFilter(new >> ScMessageProtocolCodecFactory())); >> >> chain.addLast("logger", new LoggingFilter()); >> chain.addLast("monitor", new ProtocolMonitorFilter()); >> >> ExecutorService threadPool = new OrderedThreadPoolExecutor(4); >> chain.addLast("executor", new ExecutorFilter(threadPool)); >> acceptor.setHandler(new CentralDispatchHandler()); >> >> >> -----Original Message----- >> From: Emmanuel Lecharny [mailto:[email protected]] On Behalf Of >> Emmanuel Lecharny >> Sent: Friday, May 15, 2009 1:17 AM >> To: [email protected] >> Subject: Re: memory leak? -- help needed >> >> Hi, >> >> this is very strange... If the client close the connection before the >> message is sent to him, then the session should be discarded. >> >> Could you provide the code which initialize the chain you are using ? >> >> Also could you run your test with the latest version (ie, the trunk)? We >> have fixed some bad issues lately about the way messages are handled. >> >> Thanks ! >> >> >> Yongxing Wang wrote: >> >> > In sessionClosed() method in DefaultIoFilterChain, it will do: >> > >> > s.getWriteRequestQueue().dispose(session); >> > >> > >> > There are only two classes that implements WriteRequestQueue, one is >> CloseAwareWriteQueue and another is DefaultWriteRequestQueue. >> CloseAwareWriteQueue is just a wrapper on top of DefaultWriteRequestQueue >> where the dispose() method does nothing. >> > >> > public void dispose(IoSession session) { >> > } >> > >> > >> > So I have a theory for the mem leak: >> > >> > 1: Client request something, >> > 2: Server fetch the data and call session.write(), the data got queued >> in the wirteRequestQueue. >> > 3: Before the data got actually flushed, the client drops the connection >> > 4: MINA fires sessionClosed() as expect, but the dispose method didn't >> actually clear the queue. >> > 5: Because of the session still got referenced by the Finalizer, the >> huge amount of data just hang around. >> > >> > >> > Am I right that the dispose() method should clear the queue? >> > >> > Thanks, >> > Yong >> > >> > >> > >> > >> > -----Original Message----- >> > From: Yongxing Wang >> > Sent: Thursday, May 14, 2009 9:31 AM >> > To: '[email protected]' >> > Subject: RE: memory leak? -- help needed >> > >> > Yes, I will definitely run it against one of those tools (my Jprofile >> evaluation expired, need to work it out). >> > >> > I guess last night my brain was fried and I didn't provide enough >> information. The dump analysis told me that there are still more than 7K >> NioSocketSession (referenced by java.lang.Finalizer) hanging around three >> hours after I the process out of LB. This probably related to one of my >> earlier post: >> > >> > "Looking at the source code, it is caused by DefaultIoFilterChain >> (because it has a finalize method)." -- the chain holds a reference to the >> session. >> > >> > I tried to trace the "DefaultWriteFuture" to the GC root. With 10G of >> dump data, the query never returns. >> > >> > I will continue to do more analysis. At the mean time, is there a reason >> that you have to use "finalize" method in the DefaultIoFilterChain? >> > >> > Yong >> > >> > >> > -----Original Message----- >> > From: Emmanuel Lecharny [mailto:[email protected]] On Behalf Of >> Emmanuel Lecharny >> > Sent: Wednesday, May 13, 2009 11:47 PM >> > To: [email protected] >> > Subject: Re: memory leak? -- help needed >> > >> > Yongxing Wang wrote: >> > >> >> Gurus, >> >> >> >> >> > hi, >> > >> > you should use a tool to determine if the memory leak you are >> > experiencing is part of your own code. You have plenty of them out there >> > : JProfiler, OptimizeIt, YourKit, ... >> > >> > This will give you some insight about the part of the code where this >> > memory leak is likely to be created. >> > >> > There is not much we can do with the provided information atm... >> > >> > >> >> >> -- >> -- >> cordialement, regards, >> Emmanuel Lécharny >> www.iktek.com >> directory.apache.org >> >> >> > > > -- > Regards, > Cordialement, > > Emmanuel Lécharny > www.iktek.com > -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
