Hi Kurt, Since you have the environment to reproduce this, could you run with the OpenJPA trace enabled? I'm not even sure what I'm looking for at this point. :-) But, maybe there's some tracing that would point us in the right direction.
<property name="openjpa.Log" value="DefaultLevel=TRACE"/> We have been always been on the watch for memory leaks, so this one is puzzling. You did mention blocked threads as well. At this point, you are probably hitting blocked threads due to the GC and Memory problems. But, we did recently resolve some locking situations [1] in our MetaDataRepository that could affect scaling (under extreme load). This was resolved in the 1.2.x SNAPSHOT drivers (Mike is currently pushing for a 1.2.2 release, so that should be coming soon). Does your WebService version of the problem use EntityManager transactions, or is the transaction service coming from Tomcat? And, does your WebService version also do the em.clear() and em.close()? Basically, are you using application-managed or container-managed persistence while running as a WebService? I'm very interested in helping to resolve this issue. Like I mentioned, we do a lot of memory leakage and scalability testing of the OpenJPA solution and we're not aware of anything. Although there is always the possibility that something was resolved in the 1.2.x branch after the 1.2.1 release that I'm not remembering... Thanks for your help, Kevin [1] https://issues.apache.org/jira/browse/OPENJPA-250 On Wed, Jan 13, 2010 at 9:27 PM, Kurt T Stam <[email protected]> wrote: > Thanks Kevin, > > We're enhancing at build time: > > http://svn.apache.org/repos/asf/webservices/juddi/trunk/juddi-core/pom.xml > > Yeah we've been running load tests and things are nice and stable with > Hibernate but with Openjpa we see increasing memory use, blocking threads > and then an OOM. http://issues.apache.org/jira/browse/JUDDI-267. Our > preference would be to ship with openjpa by default; but our build supports > both hibernate and openjpa. > > And yes we use openjpa 1.2.1 (latest stable version). > > --Kurt > > > Kevin Sutter wrote: > >> Interesting detective work, Kurt. Thanks. >> >> Why the WebService version of the app would behave differently as far as >> GC >> is concerned is a mystery. And, you said that plugging in Hibernate into >> this scenario, everything works okay? Very confusing. >> >> How are you performing the Entity enhancement processing? Are you >> pre-enhancing via your build process? Or, are you using the -javaagent >> mechanism? Or, are you falling back to the subclassing support within >> OpenJPA? (See [1] for more information on these questions in case they >> don't make sense.) >> >> This would be one area that is different between Hibernate and OpenJPA -- >> enhancement processing. >> >> In the Tomcat environment, you may be falling back to the subclassing >> support (which we do not recommend) and hitting a memory leak with that. >> >> You said OpenJPA 1.2.x, right? >> >> Just a couple of thoughts on the subject... >> Kevin >> >> [1] >> http://webspherepersistence.blogspot.com/2009/02/openjpa-enhancement.html >> >> >> >> On Wed, Jan 13, 2010 at 4:25 PM, Kurt T Stam <[email protected]> wrote: >> >> >> >>> The same code executed straight from a java client (inVM) shows no memory >>> leak. >>> >>> So is the fact that it is WebService significant then? What else can be >>> different? I think one thread remains up, and somehow this causes openjpa >>> not being able to clean up after itself. What can I do to debug this >>> more? I >>> can actually see in the profiler that the objects are allocated by the >>> WebService, but why aren't they cleaned up? >>> >>> Thx, >>> >>> >>> --Kurt >>> >>> >>> Kurt T Stam wrote: >>> >>> >>> >>>> Thanks Kevin, thanks for your response. >>>> >>>> I just replaced the static call by: >>>> >>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>> apiAuthToken.setAuthInfo(modelAuthToken.getAuthToken()); >>>> //MappingModelToApi.mapAuthToken(modelAuthToken, >>>> apiAuthToken); >>>> >>>> which did not make a difference. >>>> >>>> I'm wondering if the fact that my class is a webservice makes a >>>> difference. I'll try extracting it into >>>> a regular class with a main method and profile that. At least I know >>>> that >>>> I didn't forget something >>>> completely obvious.. >>>> >>>> --Kurt >>>> >>>> Kevin Sutter wrote: >>>> >>>> >>>> >>>>> Kurt, >>>>> I agree that this is very common usage of the JPA programming model. >>>>> And, >>>>> we are not aware of any memory leaks. About the only thing that jumps >>>>> out >>>>> at me is the following two lines: >>>>> >>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>> apiAuthToken); >>>>> >>>>> What do these do? Can you comment these out and see if the memory leak >>>>> still exists? Since you are passing the modelAuthToken into this >>>>> method, >>>>> I >>>>> don't know what it's doing with the reference and could it be holding >>>>> onto >>>>> something to prevent the GC from cleaning up? >>>>> >>>>> The rest of your example seems very straight forward with creating and >>>>> persisting objects. >>>>> >>>>> Kevin >>>>> >>>>> On Wed, Jan 13, 2010 at 2:09 PM, Rick Curtis <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> If you change the 1000 to something like 1000000... does your >>>>>> application >>>>>> go >>>>>> OOM? Are you running in a JSE environment? What is PersistenceManager? >>>>>> >>>>>> On Wed, Jan 13, 2010 at 2:05 PM, Kurt T Stam <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> BTW I'm running with the cache off >>>>>>> >>>>>>> <property name="openjpa.DataCache" value="false"/> >>>>>>> >>>>>>> (that turns it off right?) >>>>>>> >>>>>>> --Kurt >>>>>>> >>>>>>> >>>>>>> >>>>>>> Kurt T Stam wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> [DESCRIPTION] The code below inserts a 1000 records in the database. >>>>>>>> >>>>>>>> for (int i=1; i<1000; i++) { >>>>>>>> EntityManager em = PersistenceManager.getEntityManager(); >>>>>>>> EntityTransaction tx = em.getTransaction(); >>>>>>>> try { >>>>>>>> tx.begin(); >>>>>>>> // Generate auth token and store it! >>>>>>>> String authInfo = AUTH_TOKEN_PREFIX + UUID.randomUUID(); >>>>>>>> org.apache.juddi.model.AuthToken modelAuthToken = new >>>>>>>> org.apache.juddi.model.AuthToken(); >>>>>>>> if (authInfo != null) { >>>>>>>> modelAuthToken.setAuthToken(authInfo); >>>>>>>> modelAuthToken.setCreated(new Date()); >>>>>>>> modelAuthToken.setLastUsed(new Date()); >>>>>>>> modelAuthToken.setAuthorizedName(publisherId); >>>>>>>> modelAuthToken.setNumberOfUses(0); >>>>>>>> modelAuthToken.setTokenState(AUTHTOKEN_ACTIVE); >>>>>>>> em.persist(modelAuthToken); >>>>>>>> } >>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>> apiAuthToken); >>>>>>>> tx.commit(); >>>>>>>> } finally { >>>>>>>> if (tx.isActive()) { >>>>>>>> tx.rollback(); >>>>>>>> } >>>>>>>> em.clear(); >>>>>>>> em.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> [ISSUE] >>>>>>>> After it leaving this code I end up with a 1000 >>>>>>>> org.apache.juddi.model.AuthToken objects in memory. I've been using >>>>>>>> the >>>>>>>> profiler, and these objects cannot be garbage collected. >>>>>>>> >>>>>>>> This seems to be pretty the most common use case of using an >>>>>>>> OR-mapping >>>>>>>> tool, so I find it hard to believe openjpa has a memory leak here. >>>>>>>> Does >>>>>>>> anyone see what I'm doing wrong? Or can someone point me to an >>>>>>>> example >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> that >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>>> does not exhibit this behavior? BTW same code using hibernate does >>>>>>> not >>>>>>> >>>>>>> >>>>>>>> accumulate these objects. >>>>>>>> >>>>>>>> We're using openjpa 1.2.1. >>>>>>>> >>>>>>>> >>>>>>>> Thx, >>>>>>>> >>>>>>>> >>>>>>>> Kurt >>>>>>>> >>>>>>>> Apache jUDDI. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Rick >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> > >
