Rick, Thanks for following up.
After you email I've looked again at the hprof dump and the number of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is 1. I've also noticed the heap has 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they are referenced from a finalizer; thus in queue to be GC. So, it would argue, the leak does not come from there. I'll give it a ry using 2.2.0 to see if that makes a difference. Thanks again. Alejandro On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <curti...@gmail.com> wrote: > After digging into the heapdump, I'm fairly certain that somewhere in your > application EntityManagerFactories are being created, yet they are never > closed. I know you pointed me at your JPA service source code, but there > must be something else going on. For giggles you could move to version > 2.2.0, where we made some changes[1] to better tolerate applications that > fail to close EMFs. The JIRA talks about additional diagnostics, but we > just changed a bit of our code to use WeakReferences in hopes of avoiding > these sorts of OOMs. If moving to 2.2.0 makes your problem go away, you'll > need to look elsewhere in your code for EMF creation. Hope this helps. > > Thanks, > Rick > > > [1] https://issues.apache.org/jira/browse/OPENJPA-2042 > > > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tuc...@gmail.com > >wrote: > > > Hi Rick, a hprof dump is available at > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz > > > > Thank you very much for looking into this. > > > > Alejandro > > > > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <curti...@gmail.com> wrote: > > > > > Can you post a heapdump somewhere for review? > > > > > > Thanks, > > > Rick > > > > > > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur <tuc...@gmail.com > > > >wrote: > > > > > > > Rick, > > > > > > > > We create a single EMF: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160 > > > > > > > > And we use create/use/dispose all EMs using a command patter: > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209 > > > > > > > > Any other idea/hint? > > > > > > > > Thanks > > > > > > > > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis <curti...@gmail.com> > > > wrote: > > > > > > > > > Alejandro - > > > > > > > > > > It appears that your application is creating > EntityManagerFactories, > > > yet > > > > > they are never getting closed. > > > > > > > > > > Thanks, > > > > > Rick > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur < > > tuc...@gmail.com > > > > > >wrote: > > > > > > > > > > > Rick, by looking at the other thread memory analyzer image, the > > > issues > > > > > seem > > > > > > to be different. > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis <curti...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Alejandro -- > > > > > > > > > > > > > > Is the problem that you're asking about the same as Hovan? If > so, > > > > lets > > > > > > > consolidate this conversation down to a single thread. > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html > > > > > > > > > > > > > > Thanks, > > > > > > > Rick > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter < > > kwsut...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Alejandro, > > > > > > > > It looks like something is holding onto references... How > are > > > you > > > > > > > > enhancing your classes? At build time? Or, some other > > > mechanism? > > > > > > > > > > > > > > > > Does this memory leak happen with any of your configured > > > databases? > > > > > > Or, > > > > > > > > just select ones? > > > > > > > > > > > > > > > > This shouldn't make a difference, but specifying your Entity > > > > classes > > > > > > via > > > > > > > > both <class> and the openjpa.MetaDataFactory property is > > > overkill. > > > > > The > > > > > > > > <class> elements are sufficient and is the standard, so I'd > > > suggest > > > > > > > > sticking with that mechanism and removing the > > > > openjpa.MetaDataFactory > > > > > > > > property. > > > > > > > > > > > > > > > > We do considerable memory leak testing both as a standalone > JPA > > > > > > provider > > > > > > > as > > > > > > > > well as the JPA provider for WebSphere Application Server. > We > > > > could > > > > > > have > > > > > > > > missed something due to your configuration requirements, but > > this > > > > > looks > > > > > > > to > > > > > > > > be a pretty major leak... > > > > > > > > > > > > > > > > Can you narrow down or simplify your scenario to the point of > > > > > sharing a > > > > > > > > project that demonstrates the leak? Or, maybe there is > memory > > > data > > > > > > dumps > > > > > > > > that can be shared to help diagnose the issue? > > > > > > > > > > > > > > > > Kevin > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur < > > > > > tuc...@gmail.com > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > And .... I'm back..... > > > > > > > > > > > > > > > > > > It seems that removing runtime enhancement has some effect, > > but > > > > the > > > > > > > heap > > > > > > > > > keeps building up in the same spot. > > > > > > > > > > > > > > > > > > The following link is show where is the memory sink > > > > > > > > > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png > > > > > > > > > > > > > > > > > > I'd appreciate any ideas/suggestions. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > Alejandro > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter < > > > > kwsut...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > It shoudn't... But, with that flag set as "supported", > we > > > > can't > > > > > be > > > > > > > > sure > > > > > > > > > > that the entities were actually enhanced at build time. > By > > > > > setting > > > > > > > > this > > > > > > > > > > property to "unsupported" or removing it altogether, then > > if > > > an > > > > > > > > > unenhanced > > > > > > > > > > entity class is loaded, we'll log a message and quit > > > > processing. > > > > > > So, > > > > > > > > > let's > > > > > > > > > > go this route first and then we'll deal with any > potential > > > > memory > > > > > > > leak. > > > > > > > > > > > > > > > > > > > > Thanks, Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur < > > > > > > > tuc...@gmail.com > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > I'm in the process of verifying if Kevin's suggestion > > fixes > > > > the > > > > > > > > > problem. > > > > > > > > > > > > > > > > > > > > > > While making the necessary changes, I've notice we were > > > > already > > > > > > > > > enhancing > > > > > > > > > > > the classes at build time. > > > > > > > > > > > > > > > > > > > > > > If the classes were enhanced at build time, still would > > be > > > we > > > > > see > > > > > > > the > > > > > > > > > > leak > > > > > > > > > > > if the runtime enhancement flag is enabled? > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur < > > > > > > > tuc...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks Kevin, will try that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter < > > > > > > kwsut...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Don't use this: > > > > > > > > > > > >> > > > > > > > > > > > >> <property name="openjpa.RuntimeUnenhancedClasses" > > > > > > > > > value="supported"/> > > > > > > > > > > > >> > > > > > > > > > > > >> That is probably your issue. Reference this page > for > > > more > > > > > > > > > > information: > > > > > > > > > > > >> http://openjpa.apache.org/entity-enhancement.html > > > > > > > > > > > >> > > > > > > > > > > > >> Good luck, Kevin > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro Abdelnur > < > > > > > > > > > tuc...@gmail.com > > > > > > > > > > > >> >wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> > Hi Rick, > > > > > > > > > > > >> > > > > > > > > > > > > >> > Thanks for looking into this, the persistence.xml > > file > > > > we > > > > > > are > > > > > > > > > using > > > > > > > > > > > is: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml > > > > > > > > > > > >> > > > > > > > > > > > > >> > The EMF is instatiated here: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156 > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > Thanks again. > > > > > > > > > > > >> > > > > > > > > > > > > >> > Alejandro > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis < > > > > > > > curti...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > Can we see the contents of your persistence.xml > > > file? > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro > > Abdelnur < > > > > > > > > > > > tuc...@gmail.com> > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in > Oozie. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be holding a > > > large > > > > > > amount > > > > > > > > of > > > > > > > > > > > memory > > > > > > > > > > > >> > via > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) instance > > > > > variable > > > > > > > > > keeping > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Class # Objects Used > > > Heap > > > > > > > > Retained > > > > > > > > > > > Heap > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 > > 304,320 > > > > > > > > > > 904,410,184 > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 > > > 1,223,680 > > > > > > > > > > 105,073,624 > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory and > > we > > > > are > > > > > > > > closing > > > > > > > > > > all > > > > > > > > > > > >> > > > EntityManager instances correctly (via a > command > > > > > > pattern). > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > All caching settings are default ones. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Any hint where to look would be appreciated > it. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Thanks > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Alejandro > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > >> > > *Rick Curtis* > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > -- > > > *Rick Curtis* > > > > > > > > > -- > *Rick Curtis* >
