rick, thanks for following up, i've already scanned oozie source and we create a single EMF and use it for the lifetime of the process, so i'm discarding this as the root cause at the moment. the intereesting thing is that this has been running without any issues for 2+ years in heavy loaded prod systems. and still is running without any issues in all but a couple of system, i'm trying to identify any app usage pattern that could gove me a clue on something diff we ate doing with openjpa conns in those setups.
thx Alejandro (phone typing) On Jun 10, 2013, at 7:26, Rick Curtis <curti...@gmail.com> wrote: > No reason why I suggested 2.2.0 as opposed to 2.2.2. Either should work > just fine. > > I'd be more concerned that something is happening in your application that > you don't fully understand. Even if you move to 2.2.x, you should really > spend some time to figure out why these EMFs aren't being closed. EMF > creation is fairly expensive and if your application is performance > sensitive, you might want to figure out what is going on. > > I am glad we are able to point in the direction of the leak though. Good > luck! > > Thanks, > Rick > > > On Fri, Jun 7, 2013 at 6:37 PM, Alejandro Abdelnur <tuc...@gmail.com> wrote: > >> Rick, >> >> Following up on this, it seems that using 2.2.0 is fixing the OOM issues. >> >> BTW, any reason you sugested 2.2.0 instead 2.2.2? >> >> Thanks. >> >> >> On Tue, May 28, 2013 at 9:33 AM, Rick Curtis <curti...@gmail.com> wrote: >> >>> You are reading too far into my analysis. The leak is NOT >>> EntityManagerFactoryImpl instances, it is a data structure >>> (MappingRepository) that gets created as a result of creating an EMF. The >>> MappingRepository is statically referenced elsewhere in our code and if >> you >>> fail to close all EntityManagerFactoryImpl instances, the >> MappingRepository >>> instances don't get cleaned up. >>> >>> Thanks, >>> Rick >>> >>> >>> On Tue, May 28, 2013 at 11:24 AM, Alejandro Abdelnur <tuc...@gmail.com >>>> wrote: >>> >>>> Rick, >>>> >>>> Thanks for following up. >>>> >>>> After you email I've looked again at the hprof dump and the number >>>> of org.apache.openjpa.persistence.EntityManagerFactoryImpl isntances is >>> 1. >>>> I've also noticed the heap has >>>> 500+ org.apache.openjpa.persistence.EntityManagerImpl instances, but >> they >>>> are referenced from a finalizer; thus in queue to be GC. So, it would >>>> argue, the leak does not come from there. >>>> >>>> I'll give it a ry using 2.2.0 to see if that makes a difference. >>>> >>>> Thanks again. >>>> >>>> Alejandro >>>> >>>> >>>> >>>> On Fri, May 24, 2013 at 12:05 PM, Rick Curtis <curti...@gmail.com> >>> wrote: >>>> >>>>> After digging into the heapdump, I'm fairly certain that somewhere in >>>> your >>>>> application EntityManagerFactories are being created, yet they are >>> never >>>>> closed. I know you pointed me at your JPA service source code, but >>> there >>>>> must be something else going on. For giggles you could move to >> version >>>>> 2.2.0, where we made some changes[1] to better tolerate applications >>> that >>>>> fail to close EMFs. The JIRA talks about additional diagnostics, but >> we >>>>> just changed a bit of our code to use WeakReferences in hopes of >>> avoiding >>>>> these sorts of OOMs. If moving to 2.2.0 makes your problem go away, >>>> you'll >>>>> need to look elsewhere in your code for EMF creation. Hope this >> helps. >>>>> >>>>> Thanks, >>>>> Rick >>>>> >>>>> >>>>> [1] https://issues.apache.org/jira/browse/OPENJPA-2042 >>>>> >>>>> >>>>> On Fri, May 24, 2013 at 10:49 AM, Alejandro Abdelnur < >> tuc...@gmail.com >>>>>> wrote: >>>>> >>>>>> Hi Rick, a hprof dump is available at >>>>>> http://people.apache.org/~tucu/tmp/openjpaleak.hprof.tar.gz >>>>>> >>>>>> Thank you very much for looking into this. >>>>>> >>>>>> Alejandro >>>>>> >>>>>> >>>>>> On Fri, May 24, 2013 at 7:46 AM, Rick Curtis <curti...@gmail.com> >>>> wrote: >>>>>> >>>>>>> Can you post a heapdump somewhere for review? >>>>>>> >>>>>>> Thanks, >>>>>>> Rick >>>>>>> >>>>>>> >>>>>>> On Thu, May 23, 2013 at 6:53 PM, Alejandro Abdelnur < >>>> tuc...@gmail.com >>>>>>>> wrote: >>>>>>> >>>>>>>> Rick, >>>>>>>> >>>>>>>> We create a single EMF: >> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L160 >>>>>>>> >>>>>>>> And we use create/use/dispose all EMs using a command patter: >> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L209 >>>>>>>> >>>>>>>> Any other idea/hint? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> On Wed, May 22, 2013 at 11:32 AM, Rick Curtis < >>> curti...@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Alejandro - >>>>>>>>> >>>>>>>>> It appears that your application is creating >>>>> EntityManagerFactories, >>>>>>> yet >>>>>>>>> they are never getting closed. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Rick >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, May 22, 2013 at 12:40 PM, Alejandro Abdelnur < >>>>>> tuc...@gmail.com >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Rick, by looking at the other thread memory analyzer image, >>> the >>>>>>> issues >>>>>>>>> seem >>>>>>>>>> to be different. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, May 20, 2013 at 9:47 AM, Rick Curtis < >>>> curti...@gmail.com >>>>>> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Alejandro -- >>>>>>>>>>> >>>>>>>>>>> Is the problem that you're asking about the same as >> Hovan? >>> If >>>>> so, >>>>>>>> lets >>>>>>>>>>> consolidate this conversation down to a single thread. >>> http://openjpa.208410.n2.nabble.com/OOM-caused-by-OpenJPA-td7583869.html >> http://openjpa.208410.n2.nabble.com/OOM-with-JDBCConfigurationImpl-holding-1-GB-of-heap-td7583862.html >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Rick >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, May 20, 2013 at 11:38 AM, Kevin Sutter < >>>>>> kwsut...@gmail.com >>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Alejandro, >>>>>>>>>>>> It looks like something is holding onto references... >>> How >>>>> are >>>>>>> you >>>>>>>>>>>> enhancing your classes? At build time? Or, some other >>>>>>> mechanism? >>>>>>>>>>>> >>>>>>>>>>>> Does this memory leak happen with any of your >> configured >>>>>>> databases? >>>>>>>>>> Or, >>>>>>>>>>>> just select ones? >>>>>>>>>>>> >>>>>>>>>>>> This shouldn't make a difference, but specifying your >>>> Entity >>>>>>>> classes >>>>>>>>>> via >>>>>>>>>>>> both <class> and the openjpa.MetaDataFactory property >> is >>>>>>> overkill. >>>>>>>>> The >>>>>>>>>>>> <class> elements are sufficient and is the standard, so >>> I'd >>>>>>> suggest >>>>>>>>>>>> sticking with that mechanism and removing the >>>>>>>> openjpa.MetaDataFactory >>>>>>>>>>>> property. >>>>>>>>>>>> >>>>>>>>>>>> We do considerable memory leak testing both as a >>> standalone >>>>> JPA >>>>>>>>>> provider >>>>>>>>>>> as >>>>>>>>>>>> well as the JPA provider for WebSphere Application >>> Server. >>>>> We >>>>>>>> could >>>>>>>>>> have >>>>>>>>>>>> missed something due to your configuration >> requirements, >>>> but >>>>>> this >>>>>>>>> looks >>>>>>>>>>> to >>>>>>>>>>>> be a pretty major leak... >>>>>>>>>>>> >>>>>>>>>>>> Can you narrow down or simplify your scenario to the >>> point >>>> of >>>>>>>>> sharing a >>>>>>>>>>>> project that demonstrates the leak? Or, maybe there is >>>>> memory >>>>>>> data >>>>>>>>>> dumps >>>>>>>>>>>> that can be shared to help diagnose the issue? >>>>>>>>>>>> >>>>>>>>>>>> Kevin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, May 17, 2013 at 4:50 PM, Alejandro Abdelnur < >>>>>>>>> tuc...@gmail.com >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> And .... I'm back..... >>>>>>>>>>>>> >>>>>>>>>>>>> It seems that removing runtime enhancement has some >>>> effect, >>>>>> but >>>>>>>> the >>>>>>>>>>> heap >>>>>>>>>>>>> keeps building up in the same spot. >>>>>>>>>>>>> >>>>>>>>>>>>> The following link is show where is the memory sink >>>>>>>>>>>>> >>>>>>>>>>>>> http://people.apache.org/~tucu/tmp/OpenJPALeak.png >>>>>>>>>>>>> >>>>>>>>>>>>> I'd appreciate any ideas/suggestions. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> Alejandro >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, May 10, 2013 at 10:11 AM, Kevin Sutter < >>>>>>>> kwsut...@gmail.com >>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> It shoudn't... But, with that flag set as >>> "supported", >>>>> we >>>>>>>> can't >>>>>>>>> be >>>>>>>>>>>> sure >>>>>>>>>>>>>> that the entities were actually enhanced at build >>> time. >>>>> By >>>>>>>>> setting >>>>>>>>>>>> this >>>>>>>>>>>>>> property to "unsupported" or removing it >> altogether, >>>> then >>>>>> if >>>>>>> an >>>>>>>>>>>>> unenhanced >>>>>>>>>>>>>> entity class is loaded, we'll log a message and >> quit >>>>>>>> processing. >>>>>>>>>> So, >>>>>>>>>>>>> let's >>>>>>>>>>>>>> go this route first and then we'll deal with any >>>>> potential >>>>>>>> memory >>>>>>>>>>> leak. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 10, 2013 at 12:05 PM, Alejandro >> Abdelnur >>> < >>>>>>>>>>> tuc...@gmail.com >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm in the process of verifying if Kevin's >>> suggestion >>>>>> fixes >>>>>>>> the >>>>>>>>>>>>> problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> While making the necessary changes, I've notice >> we >>>> were >>>>>>>> already >>>>>>>>>>>>> enhancing >>>>>>>>>>>>>>> the classes at build time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If the classes were enhanced at build time, still >>>> would >>>>>> be >>>>>>> we >>>>>>>>> see >>>>>>>>>>> the >>>>>>>>>>>>>> leak >>>>>>>>>>>>>>> if the runtime enhancement flag is enabled? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, May 9, 2013 at 9:15 AM, Alejandro >> Abdelnur >>> < >>>>>>>>>>> tuc...@gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Kevin, will try that. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 8:59 AM, Kevin Sutter < >>>>>>>>>> kwsut...@gmail.com >>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Don't use this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> <property >>> name="openjpa.RuntimeUnenhancedClasses" >>>>>>>>>>>>> value="supported"/> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> That is probably your issue. Reference this >>> page >>>>> for >>>>>>> more >>>>>>>>>>>>>> information: >>> http://openjpa.apache.org/entity-enhancement.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Good luck, Kevin >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 10:17 AM, Alejandro >>>> Abdelnur >>>>> < >>>>>>>>>>>>> tuc...@gmail.com >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Rick, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for looking into this, the >>>> persistence.xml >>>>>> file >>>>>>>> we >>>>>>>>>> are >>>>>>>>>>>>> using >>>>>>>>>>>>>>> is: >> https://github.com/apache/oozie/blob/trunk/core/src/main/resources/META-INF/persistence.xml >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The EMF is instatiated here: >> https://github.com/apache/oozie/blob/trunk/core/src/main/java/org/apache/oozie/service/JPAService.java#L156 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks again. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Alejandro >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, May 9, 2013 at 6:06 AM, Rick Curtis >> < >>>>>>>>>>> curti...@gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Can we see the contents of your >>>> persistence.xml >>>>>>> file? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, May 8, 2013 at 6:34 PM, Alejandro >>>>>> Abdelnur < >>>>>>>>>>>>>>> tuc...@gmail.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We are seeing this using OpenJPA 2.1.0 >> in >>>>> Oozie. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> A JDBCConfigurationImpl seems to be >>> holding >>>> a >>>>>>> large >>>>>>>>>> amount >>>>>>>>>>>> of >>>>>>>>>>>>>>> memory >>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>>> its dataCacheTimeout (then listeners) >>>> instance >>>>>>>>> variable >>>>>>>>>>>>> keeping >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Class # Objects >>>> Used >>>>>>> Heap >>>>>>>>>>>> Retained >>>>>>>>>>>>>>> Heap >>>>> ------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> o.a.o.j.meta.MappingRepository 1,902 >>>>>> 304,320 >>>>>>>>>>>>>> 904,410,184 >>>>>>>>>>>>>>>>>>>> o.a.o.j.meta.ClassMapping 3,824 >>>>>>> 1,223,680 >>>>>>>>>>>>>> 105,073,624 >>>>> ------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We are using a single >> EntityManagerFactory >>>> and >>>>>> we >>>>>>>> are >>>>>>>>>>>> closing >>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>> EntityManager instances correctly (via a >>>>> command >>>>>>>>>> pattern). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> All caching settings are default ones. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Any hint where to look would be >>> appreciated >>>>> it. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Alejandro >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> *Rick Curtis* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Rick Curtis* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Rick Curtis* >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Rick Curtis* >>>>> >>>>> >>>>> >>>>> -- >>>>> *Rick Curtis* >>> >>> >>> >>> -- >>> *Rick Curtis* > > > > -- > *Rick Curtis*
