Hi Andrus, did you log this as a bug? And do you need anything else from me?
At the moment I added a cronjob to restart the WebService every night which prevents this bug from causing any harm but I really would like to fix it properly and remove the cronjob. Thanx Ayhan Kondoz > -----Ursprüngliche Nachricht----- > Von: Andrus Adamchik [mailto:[EMAIL PROTECTED] > Gesendet: Mittwoch, 21. Februar 2007 15:20 > An: [email protected] > Betreff: Re: AW: AW: possible bug / memory leak in DispatchQueue and > EventManager? > > 30 is good. If you had a retained DataContext somewhere you'd get the > same number as the number of stuck invocations. > > 1 million of Invocations is bad. > > I guess this needs to be logged as a possible bug to investigate, > with as many details as possible on the JVM version, etc. > > Andrus > > > On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote: > > Yes seems live there are DataContext instances. > > > > http://img73.imageshack.us/img73/3068/memta6.jpg > > > > I started the GC a few times before I took this screenshot. > > As you can see there are still 30 DataContext instances. No idea > > why thought... > > > > > >> -----Ursprüngliche Nachricht----- > >> Von: Andrus Adamchik [mailto:[EMAIL PROTECTED] > >> Gesendet: Mittwoch, 21. Februar 2007 14:49 > >> An: [email protected] > >> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and > >> EventManager? > >> > >> Do you see DataContext instances in the memory profile? I wonder how > >> many of those do you have, as those are the only Cayenne objects that > >> have hard reference to the listeners. > >> > >> Andrus > >> > >> > >> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote: > >> > >>> Hi, > >>> > >>> the JVM has 1 GB of memory and i already tried to run the GC > >>> manually but it doesn't make a difference. I can start the GC from > >>> the profiler tool but the number of instances does not change. > >>> I guess that there are some other references to this objects so > >>> that the weak ref. does not expire. > >>> > >>> And there are no EventManager exception in the log files. > >>> > >>> > >>> ayhan > >>> > >>>> -----Ursprüngliche Nachricht----- > >>>> Von: Andrus Adamchik [mailto:[EMAIL PROTECTED] > >>>> Gesendet: Mittwoch, 21. Februar 2007 14:17 > >>>> An: [email protected] > >>>> Betreff: Re: possible bug / memory leak in DispatchQueue and > >>>> EventManager? > >>>> > >>>> Seems like your assessment of the EventManager leaking is correct. > >>>> > >>>> Now the cause is not that clear. A shot in the dark - this is > >>>> due to > >>>> a combination of lots of spare memory in JVM (so weak references > >>>> are > >>>> not collected fast enough) and slow custom 'equals' and 'hashCode' > >>>> methods in invocation. > >>>> > >>>> How much heap size do you have in your server? Also can you try to > >>>> add a request filter that would do 'System.gc()' on every few > >>>> thousands requests, and see if it makes any difference. > >>>> > >>>> Also - do you see any EventManager exceptions in the logs? > >>>> > >>>> Andrus > >>>> > >>>> > >>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote: > >>>>> Hi, > >>>>> > >>>>> > >>>>> > >>>>> i think i found a possible bug / memory leak within the > >>>>> DispatchQueue and EventManager. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> First a little bit about my setup: > >>>>> > >>>>> > >>>>> > >>>>> I have 3 servers. Each server runs an axis service. The service > >>>>> uses cayenne 1.2.1 to connect to a database. It reads customer and > >>>>> account information from the DB etc.. > >>>>> > >>>>> The servers are using cayenne's shared caching with javagroups as > >>>>> the messaging service so that changes made from one server are > >>>>> dispatched to the other servers. > >>>>> > >>>>> > >>>>> > >>>>> The avarage connections per second is somewhere around 4-5. > >>>>> > >>>>> > >>>>> > >>>>> However I have a very strange problem with this setup so I startet > >>>>> to search for the reason. The problem is that with nearly constant > >>>>> and unchanging usage the load of each server increases over time. > >>>>> To further test this I created a test server with a similar setup. > >>>>> There I created a test program that creates totally constant > >>>>> usage. > >>>>> But even with the unchanging usage the load of the server is > >>>>> increasing until the cpu load is so high that the requests can not > >>>>> be processed anymore. > >>>>> > >>>>> > >>>>> > >>>>> I installed a java profiler to trying to pinpoint the location of > >>>>> this error and this is what I found out. > >>>>> > >>>>> > >>>>> > >>>>> I let the server run for 24 hours and then stopped the program > >>>>> which creates the test usage. > >>>>> > >>>>> But even while the server was idle there where still a lot of > >>>>> instances in the java heap after the GC run. > >>>>> > >>>>> > >>>>> > >>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg > >>>>> > >>>>> > >>>>> > >>>>> Please note the HashMap, WeakReference and Invocation counts. I > >>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am > >>>>> not sure. > >>>>> > >>>>> > >>>>> > >>>>> Now the following image shows the cpu profile with incoming > >>>>> connections. > >>>>> > >>>>> > >>>>> > >>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg > >>>>> > >>>>> > >>>>> > >>>>> As you can see 58% of the cpu time is used within HashSet.add(). > >>>>> > >>>>> > >>>>> > >>>>> So when I consider the two facts i think that there might be a > >>>>> possible problem with the EventManager. The first table tells us > >>>>> that there are over 1 million instances of HashSet's and cayenne > >>>>> Invocations. So it seems like the set's within the DispatchQueue > >>>>> are not recycled properly so that the object count rises over time > >>>>> which result in extremly high processtime when trying to add new > >>>>> HashSet's. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Thanks > >>>>> > >>>>> Ayhan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Ayhan Kondoz > >>>>> > >>>>> > >>>>> > >>>>> Software-Entwicklung > >>>>> > >>>>> > >>>>> > >>>>> ------------------------------------------------------------------ > >>>>> -- > >>>>> -- > >>>>> ------------ > >>>>> > >>>>> Telefon: +49 (0) 40 513 06 616 > >>>>> > >>>>> Telefax: +49 (0) 40 513 06 998 616 > >>>>> > >>>>> E-Mail: [EMAIL PROTECTED] > >>>>> <mailto:[EMAIL PROTECTED] > >>>>> ag.de> > >>>>> > >>>>> Website: http://www.freenet.de <http://www.freenet.de/> ; http:// > >>>>> www.freenet-ag.de <http://www.freenet-ag.de/> > >>>>> > >>>>> ------------------------------------------------------------------ > >>>>> -- > >>>>> -- > >>>>> ------------ > >>>>> > >>>>> freenet.de AG > >>>>> > >>>>> Deelbögenkamp 4c > >>>>> > >>>>> 22297 Hamburg > >>>>> > >>>>> ------------------------------------------------------------------ > >>>>> -- > >>>>> -- > >>>>> ------------ > >>>>> > >>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma > >>>>> > >>>>> Vorstand: Eckhard Spoerr (Vors.), > >>>>> Axel Krieger, Stephan Esch, Eric Berger > >>>>> > >>>>> Amtsgericht Hamburg HRB 74048 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Diese Information ist ausschließlich für die adressierte Person > >>>>> oder Organisation bestimmt und könnte vertrauliches und/oder > >>>>> privilegiertes Material enthalten. Personen oder Organisationen, > >>>>> für die diese Information nicht bestimmt ist, ist es nicht > >>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten, > >>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen, > >>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese > >>>>> Nachricht > >>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem > >>>>> Absender in > >>>>> Verbindung zu setzen und das Material von Ihrem Computer zu > >>>>> löschen. > >>>>> > >>>>> > >>>>> > >>>>> The information transmitted is intended only for the person or > >>>>> entity to which it is addressed and may contain confidential > >>>>> and/or > >>>>> privileged material. Any review, retransmission, dissemination or > >>>>> other use of, or taking of any action in reliance upon, this > >>>>> information by persons or entities other than the intended > >>>>> recipient is prohibited. If you received this in error, please > >>>>> contact the sender and delete the material from any computer. > >>>>> > >>>>> > >>>>> > >>> > >>> > > > >
