After digging into the heapdump, I'm fairly certain that somewhere in your application EntityManagerFactories are being created, yet they are never closed. I know you pointed me at your JPA service source code, but there must be something else going on. For giggles you could move to version 2.2.0, where we made some changes[1] to better tolerate applications that fail to close EMFs. The JIRA talks about additional diagnostics, but we just changed a bit of our code to use WeakReferences in hopes of avoiding these sorts of OOMs. If moving to 2.2.0 makes your problem go away, you'll need to look elsewhere in your code for EMF creation. Hope this helps.
Thanks, Rick [1] https://issues.apache.org/jira/browse/OPENJPA-2042 On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tuc...@gmail.com>wrote: > Hi Rick, a hprof dump is available at > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz > > Thank you very much for looking into this. > > Alejandro > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <curti...@gmail.com> wrote: > > > Can you post a heapdump somewhere for review? > > > > Thanks, > > Rick > > > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tuc...@gmail.com > > >wrote: > > > > > Rick, > > > > > > We create a single EMF: > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160 > > > > > > And we use create/use/dispose all EMs using a command patter: > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209 > > > > > > Any other idea/hint? > > > > > > Thanks > > > > > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <curti...@gmail.com> > > wrote: > > > > > > > Alejandro - > > > > > > > > It appears that your application is creating EntityManagerFactories, > > yet > > > > they are never getting closed. > > > > > > > > Thanks, > > > > Rick > > > > > > > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur < > tuc...@gmail.com > > > > >wrote: > > > > > > > > > Rick, by looking at the other thread memory analyzer image, the > > issues > > > > seem > > > > > to be different. > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <curti...@gmail.com> > > > wrote: > > > > > > > > > > > Alejandro -- > > > > > > > > > > > > Is the problem that you're asking about the same as Hovan? If so, > > > lets > > > > > > consolidate this conversation down to a single thread. > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html > > > > > > > > > > > > Thanks, > > > > > > Rick > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter < > kwsut...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Alejandro, > > > > > > > It looks like something is holding onto references... How are > > you > > > > > > > enhancing your classes? At build time? Or, some other > > mechanism? > > > > > > > > > > > > > > Does this memory leak happen with any of your configured > > databases? > > > > > Or, > > > > > > > just select ones? > > > > > > > > > > > > > > This shouldn't make a difference, but specifying your Entity > > > classes > > > > > via > > > > > > > both <class> and the openjpa.MetaDataFactory property is > > overkill. > > > > The > > > > > > > <class> elements are sufficient and is the standard, so I'd > > suggest > > > > > > > sticking with that mechanism and removing the > > > openjpa.MetaDataFactory > > > > > > > property. > > > > > > > > > > > > > > We do considerable memory leak testing both as a standalone JPA > > > > > provider > > > > > > as > > > > > > > well as the JPA provider for WebSphere Application Server. We > > > could > > > > > have > > > > > > > missed something due to your configuration requirements, but > this > > > > looks > > > > > > to > > > > > > > be a pretty major leak... > > > > > > > > > > > > > > Can you narrow down or simplify your scenario to the point of > > > > sharing a > > > > > > > project that demonstrates the leak? Or, maybe there is memory > > data > > > > > dumps > > > > > > > that can be shared to help diagnose the issue? > > > > > > > > > > > > > > Kevin > > > > > > > > > > > > > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur < > > > > tuc...@gmail.com > > > > > > > >wrote: > > > > > > > > > > > > > > > And .... I'm back..... > > > > > > > > > > > > > > > > It seems that removing runtime enhancement has some effect, > but > > > the > > > > > > heap > > > > > > > > keeps building up in the same spot. > > > > > > > > > > > > > > > > The following link is show where is the memory sink > > > > > > > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png > > > > > > > > > > > > > > > > I'd appreciate any ideas/suggestions. > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > Alejandro > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter < > > > kwsut...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > It shoudn't... But, with that flag set as "supported", we > > > can't > > > > be > > > > > > > sure > > > > > > > > > that the entities were actually enhanced at build time. By > > > > setting > > > > > > > this > > > > > > > > > property to "unsupported" or removing it altogether, then > if > > an > > > > > > > > unenhanced > > > > > > > > > entity class is loaded, we'll log a message and quit > > > processing. > > > > > So, > > > > > > > > let's > > > > > > > > > go this route first and then we'll deal with any potential > > > memory > > > > > > leak. > > > > > > > > > > > > > > > > > > Thanks, Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur < > > > > > > tuc...@gmail.com > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > I'm in the process of verifying if Kevin's suggestion > fixes > > > the > > > > > > > > problem. > > > > > > > > > > > > > > > > > > > > While making the necessary changes, I've notice we were > > > already > > > > > > > > enhancing > > > > > > > > > > the classes at build time. > > > > > > > > > > > > > > > > > > > > If the classes were enhanced at build time, still would > be > > we > > > > see > > > > > > the > > > > > > > > > leak > > > > > > > > > > if the runtime enhancement flag is enabled? > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur < > > > > > > tuc...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks Kevin, will try that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter < > > > > > kwsut...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Don't use this: > > > > > > > > > > >> > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses" > > > > > > > > value="supported"/> > > > > > > > > > > >> > > > > > > > > > > >> That is probably your issue. Reference this page for > > more > > > > > > > > > information: > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html > > > > > > > > > > >> > > > > > > > > > > >> Good luck, Kevin > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur < > > > > > > > > tuc...@gmail.com > > > > > > > > > > >> >wrote: > > > > > > > > > > >> > > > > > > > > > > >> > Hi Rick, > > > > > > > > > > >> > > > > > > > > > > > >> > Thanks for looking into this, the persistence.xml > file > > > we > > > > > are > > > > > > > > using > > > > > > > > > > is: > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml > > > > > > > > > > >> > > > > > > > > > > > >> > The EMF is instatiated here: > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156 > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > Thanks again. > > > > > > > > > > >> > > > > > > > > > > > >> > Alejandro > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis < > > > > > > curti...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > > Can we see the contents of your persistence.xml > > file? > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro > Abdelnur < > > > > > > > > > > tuc...@gmail.com> > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in Oozie. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a > > large > > > > > amount > > > > > > > of > > > > > > > > > > memory > > > > > > > > > > >> > via > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance > > > > variable > > > > > > > > keeping > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Class # Objects Used > > Heap > > > > > > > Retained > > > > > > > > > > Heap > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 > 304,320 > > > > > > > > > 904,410,184 > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 > > 1,223,680 > > > > > > > > > 105,073,624 > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory and > we > > > are > > > > > > > closing > > > > > > > > > all > > > > > > > > > > >> > > > EntityManager instances correctly (via a command > > > > > pattern). > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > All caching settings are default ones. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Any hint where to look would be appreciated it. > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Thanks > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Alejandro > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > -- > > > > > > > > > > >> > > *Rick Curtis* > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Rick Curtis* > > > > > > > > > > > > > > > -- > > *Rick Curtis* > > > -- *Rick Curtis*
