Hi Emmanuel, Thanks for analyzing the heapdumps.
You have mentioned two possible scenarios for this accumulation of NioSocketSession objects. 1. - the clients start a connection, and never close it. After finishing the testrun, I had checked number of connections to port using netstat command. There were 0 connections. So I don't think this can be the cause. I checked below workitem is mentioned in apacheDS jira. Is this going to be included in any upoming apacheDS release? http://mail-archives.apache.org/mod_mbox/directory-dev/201808.mbox/%3cjira.13144434.1520885847000.120166.1534807140...@atlassian.jira%3E 2. - the clients start a connection, close it, but the server never ditch it. In your test, can you tell me if your client is closing the connection, and how ? Our testrun client executes new threads repetitively where every thread performs below operations before terminating the thread. execute bind perform search execute unbind We are observing memory growth if we enable or disable the "execute unbind" task for the threads before termination. We did some investigation in MINA for our test run and observed that the session objects present in memory for longer time have status "Closing". We are expecting this could be because of below issue https://issues.apache.org/jira/browse/DIRMINA-1021 The fix for this issue seems included in MINA version which is bundled in apacheDS M24. Going to apacheDS M24/M25 will be time consuming process for us. Although, we will plan to upgrade in future. Emmanuel, could you comment on below queries related to MINA issues for apacheDS? Your response will help us deciding to test the fix for DIRMINA-1021. If you suggest then I will forward this mailthread on MINA forum instead. 1. Can heapmemory get full because of this issue? Or something in MINA/apacheDS will remove these accumulated CLOSING state NioSocketSession objects? Will the heapdumps show the exact same picture (top consumer hierarchy) if memory crashes because of this issue? We noticed in our heapdumps that top consumer hierarchy is almost similar to the hprof I shared with you. We observed large number of NioSocketSession objects were occupying more than 95% of memory. 2. The mailthread for DIRMINA-1021 mentions code changes as "writebuffer method should call removeNow method when IOException occurs". Could you please comment if this single line code change needed for this fix? (If you suggest, then I will forward this question on MINA forum instead.) Thanks and Regards, Pankaj Rathod On Fri, Oct 12, 2018 at 8:43 PM Emmanuel Lécharny <[email protected]> wrote: > > > Le 12/10/2018 à 22:14, Pankaj Rathod a écrit : > > Hi Emmanuel, > > > > I could finally generate dumps in hprof format. I have shared you > heapdumps > > (java.2018.10.01-11.07.22.hprof) in same folder where earlier I had sent > > PHD format dumps. > > > > This heapdump is for very small heap memory size. But you can notice in > > this dump that there are around 1818 objects of NioSocketSession class. > > > > In our system environment (and in our PHD dumps), we see around 80000 > > objects of NioSocketSession class, which combinely occupying around 95% > of > > heap memory. This is the leak suspect as per our PHD dump analysis. The > > heap analyzer tool mentions One instance of NioSocketAcceptor occupies > 95% > > memory. > > > > Please ignore the memory occupied by ACLTableGenerator class in > heapdumps. > > It has fixed size. It is a custom class developed for our need. > > Hi ! > > I just quickly checked the provided hprof file, which has 2469 > NioSession instances. > > What it means is that many LDAP conections have been initiated, and > never dropped. > > At this point, I can see 2 scenario : > > - the clients start a connection, and never close it > - the clients start a connection, close it, but the server never ditch it > > In the second case, we should check wht happens when the client close > the connection. The NioSession object should be discarded. The first > scenario is more annoying : it's perfectly possible that the client > never close the connection, so the serve rkeep the NioSession forever. > We should have a timeout for this case, which automatically closes the > connection after a configured innactivity timeout. > > In your test, can you tell me if your client is closing the connection, > and how ? Thanks ! > > -- > Emmanuel Lecharny > > Symas.com > directory.apache.org > >
