Very nice, thanks for the detective work Chathuri. Suresh
On Jun 6, 2014, at 3:55 PM, Chathuri Wimalasena <[email protected]> wrote: > With the help of OpneJPA community, I was able to solve the memory leak > issue. It is due to the way we close the entity manager. After fixing that, > we no longer see previous memory leak. Attaching the memory graphs after the > fix and before the fix. > > Thanks.. > Chathuri > > > > On Fri, May 30, 2014 at 3:50 PM, Supun Kamburugamuva <[email protected]> > wrote: > You can use Eclipse memory analyzer for finding the leak.I find it much > easier than JProfiler. > > http://www.eclipse.org/mat/ > > Thanks, > Supun.. > > > On Fri, May 30, 2014 at 3:44 PM, Chathuri Wimalasena <[email protected]> > wrote: > I did some performance testing with JConsole. Attaching two images with > caching and without. I ran getAllUserExperiments 500 times with 180 > experiments in the database. To run 500 iterations with caching is like 5 > mins but without caching it took more than a hour. In both cases memory and > cpu usages are not much difference as I feel. But it seems we have a memory > leak somewhere in our code since memory usage is increasing over the time. I > will dig more into it. > > > On Fri, May 30, 2014 at 9:30 AM, Marlon Pierce <[email protected]> wrote: > +1 very nice improvement. > On 5/29/14 2:41 PM, Saminda Wijeratne wrote: > > wow.... thats great news... Could you also monitor memory and CPU usage > > impact? See also if there's any drawbacks that'll affect us for using > > openJPA caching. > > > > > > On Thu, May 29, 2014 at 11:54 AM, Lahiru Gunathilake <[email protected]> > > wrote: > > > >> Sounds great Chathuri, I think we can tweak the caching configuration of > >> openJPA and find out the best caching configuration and publish it in wiki. > >> > >> If you can make the caching parameters configurable in > >> airavata-server.properties that would be great. > >> > >> Regards > >> Lahiru > >> > >> > >> On Thu, May 29, 2014 at 2:42 PM, Chathuri Wimalasena <[email protected] > >>> wrote: > >>> I enabled openJPA caching and the results are pretty impressive. I have > >>> 180 experiments in default derby database. I query getAllUserExperiments > >>> 100 times and average time is around 7000 - 8000 milliseconds without > >>> enabling caching. After enabling caching that number reduced to average of > >>> 400 - 500 milliseconds. > >>> > >>> > >>> On Tue, May 27, 2014 at 4:21 PM, Chathuri Wimalasena < > >>> [email protected]> wrote: > >>> > >>>> We can try openJPA caching [1] and see how it improves the performance. > >>>> This will have minimal changes to the existing code. > >>>> > >>>> [1] > >>>> http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftejb_datcacheconfig.html > >>>> > >>>> > >>>> On Tue, May 27, 2014 at 12:38 PM, Supun Kamburugamuva <[email protected] > >>>>> wrote: > >>>>> I'm not 100% percent sure on the architecture and the requirements. May > >>>>> be you should look in to Redis or Memcache. > >>>>> > >>>>> Thanks, > >>>>> Supun.. > >>>>> > >>>>> > >>>>> On Tue, May 27, 2014 at 12:35 PM, Lahiru Gunathilake <[email protected] > >>>>>> wrote: > >>>>>> I think we can implement this from the scratch and won't be a hard > >>>>>> thing to do. > >>>>>> > >>>>>> > >>>>>> On Tue, May 27, 2014 at 12:25 PM, Saminda Wijeratne < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> +1. > >>>>>>> > >>>>>>> Do you know any good frameworks which supports this? IMO the tricky > >>>>>>> part is when to know that the data in cache has expired. Jackrabbit > >>>>>>> had a > >>>>>>> feature where it does a callback whenever something gets updated in > >>>>>>> some > >>>>>>> tree path. If we restrict all registry access through a single > >>>>>>> component/layer instance we would be able to do the same by a > >>>>>>> subscription > >>>>>>> pattern. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, May 27, 2014 at 9:04 AM, Lahiru Gunathilake < > >>>>>>> [email protected]> wrote: > >>>>>>> > >>>>>>>> Hi Devs, > >>>>>>>> > >>>>>>>> In our current implementation we have large number of Experiment > >>>>>>>> retrieval and experiment storing happen in between experiment > >>>>>>>> creation and > >>>>>>>> experiment completion. We do not really parse these data-model > >>>>>>>> objects > >>>>>>>> between component and we simply parse the ids of these experiment so > >>>>>>>> every > >>>>>>>> component has to retrieve them everytime. I think programatically > >>>>>>>> this > >>>>>>>> approach looks much cleaner than parsing big objects. But to make > >>>>>>>> this more > >>>>>>>> efficient we can use a cachedRegistry implementation as another > >>>>>>>> implementation of registry and make sure we do not get objects all > >>>>>>>> the way > >>>>>>>> from the database. > >>>>>>>> > >>>>>>>> Each component can init its own cache registry object and it will > >>>>>>>> build a cache on that module and update the cache if some other > >>>>>>>> component > >>>>>>>> had changed the data-model objects. IMHO if we implement a good > >>>>>>>> caching > >>>>>>>> layer on our current data-model airavata registry will be more > >>>>>>>> efficient. > >>>>>>>> > >>>>>>>> WDYT ? > >>>>>>>> > >>>>>>>> Regards > >>>>>>>> Lahiru > >>>>>>>> > >>>>>>>> -- > >>>>>>>> System Analyst Programmer > >>>>>>>> PTI Lab > >>>>>>>> Indiana University > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> System Analyst Programmer > >>>>>> PTI Lab > >>>>>> Indiana University > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Supun Kamburugamuva > >>>>> Member, Apache Software Foundation; http://www.apache.org > >>>>> E-mail: [email protected]; Mobile: +1 812 369 6762 > >>>>> Blog: http://supunk.blogspot.com > >>>>> > >>>>> > >> > >> -- > >> System Analyst Programmer > >> PTI Lab > >> Indiana University > >> > > > > > > -- > Supun Kamburugamuva > Member, Apache Software Foundation; http://www.apache.org > E-mail: [email protected]; Mobile: +1 812 369 6762 > Blog: http://supunk.blogspot.com > > > <after_fix.png><before_fix.png>
