Rick, Following up on this, it seems that using 2.2.0 is fixing the OOM issues.
BTW, any reason you sugested 2.2.0 instead 2.2.2? Thanks. On Tue, May 28, 2013 at 9:33 AM, Rick Curtis <curti...@gmail.com> wrote: > You are reading too far into my analysis. The leak is NOT > EntityManagerFactoryImpl instances, it is a data structure > (MappingRepository) that gets created as a result of creating an EMF. The > MappingRepository is statically referenced elsewhere in our code and if you > fail to close all EntityManagerFactoryImpl instances, the MappingRepository > instances don't get cleaned up. > > Thanks, > Rick > > > On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tuc...@gmail.com > >wrote: > > > Rick, > > > > Thanks for following up. > > > > After you email I've looked again at the hprof dump and the number > > of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is > 1. > > I've also noticed the heap has > > 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but they > > are referenced from a finalizer; thus in queue to be GC. So, it would > > argue, the leak does not come from there. > > > > I'll give it a ry using 2.2.0 to see if that makes a difference. > > > > Thanks again. > > > > Alejandro > > > > > > > > On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <curti...@gmail.com> > wrote: > > > > > After digging into the heapdump, I'm fairly certain that somewhere in > > your > > > application EntityManagerFactories are being created, yet they are > never > > > closed. I know you pointed me at your JPA service source code, but > there > > > must be something else going on. For giggles you could move to version > > > 2.2.0, where we made some changes[1] to better tolerate applications > that > > > fail to close EMFs. The JIRA talks about additional diagnostics, but we > > > just changed a bit of our code to use WeakReferences in hopes of > avoiding > > > these sorts of OOMs. If moving to 2.2.0 makes your problem go away, > > you'll > > > need to look elsewhere in your code for EMF creation. Hope this helps. > > > > > > Thanks, > > > Rick > > > > > > > > > [1] https://issues.apache.org/jira/browse/OPENJPA-2042 > > > > > > > > > On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur <tuc...@gmail.com > > > >wrote: > > > > > > > Hi Rick, a hprof dump is available at > > > > http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz > > > > > > > > Thank you very much for looking into this. > > > > > > > > Alejandro > > > > > > > > > > > > On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <curti...@gmail.com> > > wrote: > > > > > > > > > Can you post a heapdump somewhere for review? > > > > > > > > > > Thanks, > > > > > Rick > > > > > > > > > > > > > > > On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur < > > tuc...@gmail.com > > > > > >wrote: > > > > > > > > > > > Rick, > > > > > > > > > > > > We create a single EMF: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160 > > > > > > > > > > > > And we use create/use/dispose all EMs using a command patter: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209 > > > > > > > > > > > > Any other idea/hint? > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 11:32 AM, Rick Curtis < > curti...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Alejandro - > > > > > > > > > > > > > > It appears that your application is creating > > > EntityManagerFactories, > > > > > yet > > > > > > > they are never getting closed. > > > > > > > > > > > > > > Thanks, > > > > > > > Rick > > > > > > > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur < > > > > tuc...@gmail.com > > > > > > > >wrote: > > > > > > > > > > > > > > > Rick, by looking at the other thread memory analyzer image, > the > > > > > issues > > > > > > > seem > > > > > > > > to be different. > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 9:47 AM, Rick Curtis < > > curti...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Alejandro -- > > > > > > > > > > > > > > > > > > Is the problem that you're asking about the same as Hovan? > If > > > so, > > > > > > lets > > > > > > > > > consolidate this conversation down to a single thread. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Rick > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter < > > > > kwsut...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Alejandro, > > > > > > > > > > It looks like something is holding onto references... > How > > > are > > > > > you > > > > > > > > > > enhancing your classes? At build time? Or, some other > > > > > mechanism? > > > > > > > > > > > > > > > > > > > > Does this memory leak happen with any of your configured > > > > > databases? > > > > > > > > Or, > > > > > > > > > > just select ones? > > > > > > > > > > > > > > > > > > > > This shouldn't make a difference, but specifying your > > Entity > > > > > > classes > > > > > > > > via > > > > > > > > > > both <class> and the openjpa.MetaDataFactory property is > > > > > overkill. > > > > > > > The > > > > > > > > > > <class> elements are sufficient and is the standard, so > I'd > > > > > suggest > > > > > > > > > > sticking with that mechanism and removing the > > > > > > openjpa.MetaDataFactory > > > > > > > > > > property. > > > > > > > > > > > > > > > > > > > > We do considerable memory leak testing both as a > standalone > > > JPA > > > > > > > > provider > > > > > > > > > as > > > > > > > > > > well as the JPA provider for WebSphere Application > Server. > > > We > > > > > > could > > > > > > > > have > > > > > > > > > > missed something due to your configuration requirements, > > but > > > > this > > > > > > > looks > > > > > > > > > to > > > > > > > > > > be a pretty major leak... > > > > > > > > > > > > > > > > > > > > Can you narrow down or simplify your scenario to the > point > > of > > > > > > > sharing a > > > > > > > > > > project that demonstrates the leak? Or, maybe there is > > > memory > > > > > data > > > > > > > > dumps > > > > > > > > > > that can be shared to help diagnose the issue? > > > > > > > > > > > > > > > > > > > > Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur < > > > > > > > tuc...@gmail.com > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > And .... I'm back..... > > > > > > > > > > > > > > > > > > > > > > It seems that removing runtime enhancement has some > > effect, > > > > but > > > > > > the > > > > > > > > > heap > > > > > > > > > > > keeps building up in the same spot. > > > > > > > > > > > > > > > > > > > > > > The following link is show where is the memory sink > > > > > > > > > > > > > > > > > > > > > > http://people.apache.org/~tucu/tmp/OpenJPALeak.png > > > > > > > > > > > > > > > > > > > > > > I'd appreciate any ideas/suggestions. > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > Alejandro > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter < > > > > > > kwsut...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > It shoudn't... But, with that flag set as > "supported", > > > we > > > > > > can't > > > > > > > be > > > > > > > > > > sure > > > > > > > > > > > > that the entities were actually enhanced at build > time. > > > By > > > > > > > setting > > > > > > > > > > this > > > > > > > > > > > > property to "unsupported" or removing it altogether, > > then > > > > if > > > > > an > > > > > > > > > > > unenhanced > > > > > > > > > > > > entity class is loaded, we'll log a message and quit > > > > > > processing. > > > > > > > > So, > > > > > > > > > > > let's > > > > > > > > > > > > go this route first and then we'll deal with any > > > potential > > > > > > memory > > > > > > > > > leak. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Kevin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2013 at 12:05 PM, Alejandro Abdelnur > < > > > > > > > > > tuc...@gmail.com > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I'm in the process of verifying if Kevin's > suggestion > > > > fixes > > > > > > the > > > > > > > > > > > problem. > > > > > > > > > > > > > > > > > > > > > > > > > > While making the necessary changes, I've notice we > > were > > > > > > already > > > > > > > > > > > enhancing > > > > > > > > > > > > > the classes at build time. > > > > > > > > > > > > > > > > > > > > > > > > > > If the classes were enhanced at build time, still > > would > > > > be > > > > > we > > > > > > > see > > > > > > > > > the > > > > > > > > > > > > leak > > > > > > > > > > > > > if the runtime enhancement flag is enabled? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 9:15 AM, Alejandro Abdelnur > < > > > > > > > > > tuc...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Kevin, will try that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter < > > > > > > > > kwsut...@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Don't use this: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> <property > name="openjpa.RuntimeUnenhancedClasses" > > > > > > > > > > > value="supported"/> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> That is probably your issue. Reference this > page > > > for > > > > > more > > > > > > > > > > > > information: > > > > > > > > > > > > > >> > http://openjpa.apache.org/entity-enhancement.html > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Good luck, Kevin > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> On Thu, May 9, 2013 at 10:17 AM, Alejandro > > Abdelnur > > > < > > > > > > > > > > > tuc...@gmail.com > > > > > > > > > > > > > >> >wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Hi Rick, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Thanks for looking into this, the > > persistence.xml > > > > file > > > > > > we > > > > > > > > are > > > > > > > > > > > using > > > > > > > > > > > > > is: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > The EMF is instatiated here: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156 > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Thanks again. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Alejandro > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Thu, May 9, 2013 at 6:06 AM, Rick Curtis < > > > > > > > > > curti...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Can we see the contents of your > > persistence.xml > > > > > file? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > On Wed, May 8, 2013 at 6:34 PM, Alejandro > > > > Abdelnur < > > > > > > > > > > > > > tuc...@gmail.com> > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > We are seeing this using OpenJPA 2.1.0 in > > > Oozie. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > A JDBCConfigurationImpl seems to be > holding > > a > > > > > large > > > > > > > > amount > > > > > > > > > > of > > > > > > > > > > > > > memory > > > > > > > > > > > > > >> > via > > > > > > > > > > > > > >> > > > its dataCacheTimeout (then listeners) > > instance > > > > > > > variable > > > > > > > > > > > keeping > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Class # Objects > > Used > > > > > Heap > > > > > > > > > > Retained > > > > > > > > > > > > > Heap > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > o.a.o.j.meta.MappingRepository 1,902 > > > > 304,320 > > > > > > > > > > > > 904,410,184 > > > > > > > > > > > > > >> > > > o.a.o.j.meta.ClassMapping 3,824 > > > > > 1,223,680 > > > > > > > > > > > > 105,073,624 > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------- > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > We are using a single EntityManagerFactory > > and > > > > we > > > > > > are > > > > > > > > > > closing > > > > > > > > > > > > all > > > > > > > > > > > > > >> > > > EntityManager instances correctly (via a > > > command > > > > > > > > pattern). > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > All caching settings are default ones. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Any hint where to look would be > appreciated > > > it. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Thanks > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Alejandro > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > > > >> > > *Rick Curtis* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Rick Curtis* > > > > > > > > > > > > > > > > > > > > > -- > > > *Rick Curtis* > > > > > > > > > -- > *Rick Curtis* >
