We must have been crossing in the ether... :-) No need for the trace, I can do that now. Thanks anyway!
Kevin On Thu, Jan 14, 2010 at 1:55 PM, Kurt T Stam <[email protected]> wrote: > OK :). > > 1. Unzip tomcat-1.0.SNAPSHOT.zip > > 2. cd tomcat-1.0.SNAPSHOT/bin > > 3. ./start.sh > > Now, to send in a request open your browser to url: > > > http://localhost:8080/leak/test > > Hit refresh to send in another request. > > I'll try getting you a trace log. Which packages? Just > "org.apache.openjpa"? > > Cheers, > > --Kurt > > > Kevin Sutter wrote: > >> Hi Kurt, >> >> These instructions, of course, assume that I know something about Tomcat. >> This would not be a good assumption... :-) If you really want me to see >> this in action, I will need more information on how to start Tomcat and >> how >> to load applications. In the mean time, a trace log of OpenJPA would be >> an >> excellent start. >> >> Thanks! >> Kevin >> >> On Thu, Jan 14, 2010 at 11:30 AM, Kurt T Stam <[email protected]> >> wrote: >> >> >> >>> Hi Kevin, >>> >>> I tried adding the javaagent stuff, and it still ends up leaking. I >>> extracted the code into a tiny >>> war with just one servlet which persist the authToken. The build creates >>> a >>> Tomcat distro backed by a >>> derby database. You can download it from: >>> >>> http://people.apache.org/~kstam/leak.tgz<http://people.apache.org/%7Ekstam/leak.tgz> >>> <http://people.apache.org/%7Ekstam/leak.tgz> >>> >>> >>> You can build it with: mvn install, you can find the final tomcat in >>> tomcat/target/tomcat-1.0.SNAPSHOT.zip >>> >>> or if you don't want to build it: >>> http://people.apache.org/~kstam/tomcat-1.0.SNAPSHOT.zip<http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> >>> <http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> >>> >>> >>> After you bring up tomcat it persistent an authToken when you hit >>> http://localhost:8080/leak/test >>> >>> In the profiler I see an org.apache.juddi.AuthToken left behind for each >>> request. >>> >>> Hopefully this little example will pinpoint what's going on. >>> >>> Thx Kev for sticking with me :) >>> >>> --Kurt >>> >>> >>> >>> Kevin Sutter wrote: >>> >>> >>> >>>> Hi Kurt, >>>> Since you have the environment to reproduce this, could you run with the >>>> OpenJPA trace enabled? I'm not even sure what I'm looking for at this >>>> point. :-) But, maybe there's some tracing that would point us in the >>>> right direction. >>>> >>>> <property name="openjpa.Log" value="DefaultLevel=TRACE"/> >>>> >>>> We have been always been on the watch for memory leaks, so this one is >>>> puzzling. You did mention blocked threads as well. At this point, you >>>> are >>>> probably hitting blocked threads due to the GC and Memory problems. >>>> But, >>>> we >>>> did recently resolve some locking situations [1] in our >>>> MetaDataRepository >>>> that could affect scaling (under extreme load). This was resolved in >>>> the >>>> 1.2.x SNAPSHOT drivers (Mike is currently pushing for a 1.2.2 release, >>>> so >>>> that should be coming soon). >>>> >>>> Does your WebService version of the problem use EntityManager >>>> transactions, >>>> or is the transaction service coming from Tomcat? And, does your >>>> WebService >>>> version also do the em.clear() and em.close()? Basically, are you using >>>> application-managed or container-managed persistence while running as a >>>> WebService? >>>> >>>> I'm very interested in helping to resolve this issue. Like I mentioned, >>>> we >>>> do a lot of memory leakage and scalability testing of the OpenJPA >>>> solution >>>> and we're not aware of anything. Although there is always the >>>> possibility >>>> that something was resolved in the 1.2.x branch after the 1.2.1 release >>>> that >>>> I'm not remembering... >>>> >>>> Thanks for your help, >>>> Kevin >>>> >>>> [1] https://issues.apache.org/jira/browse/OPENJPA-250 >>>> >>>> On Wed, Jan 13, 2010 at 9:27 PM, Kurt T Stam <[email protected]> >>>> wrote: >>>> >>>> >>>> >>>> >>>> >>>>> Thanks Kevin, >>>>> >>>>> We're enhancing at build time: >>>>> >>>>> >>>>> >>>>> http://svn.apache.org/repos/asf/webservices/juddi/trunk/juddi-core/pom.xml >>>>> >>>>> Yeah we've been running load tests and things are nice and stable with >>>>> Hibernate but with Openjpa we see increasing memory use, blocking >>>>> threads >>>>> and then an OOM. http://issues.apache.org/jira/browse/JUDDI-267. Our >>>>> preference would be to ship with openjpa by default; but our build >>>>> supports >>>>> both hibernate and openjpa. >>>>> >>>>> And yes we use openjpa 1.2.1 (latest stable version). >>>>> >>>>> --Kurt >>>>> >>>>> >>>>> Kevin Sutter wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Interesting detective work, Kurt. Thanks. >>>>>> >>>>>> Why the WebService version of the app would behave differently as far >>>>>> as >>>>>> GC >>>>>> is concerned is a mystery. And, you said that plugging in Hibernate >>>>>> into >>>>>> this scenario, everything works okay? Very confusing. >>>>>> >>>>>> How are you performing the Entity enhancement processing? Are you >>>>>> pre-enhancing via your build process? Or, are you using the >>>>>> -javaagent >>>>>> mechanism? Or, are you falling back to the subclassing support within >>>>>> OpenJPA? (See [1] for more information on these questions in case >>>>>> they >>>>>> don't make sense.) >>>>>> >>>>>> This would be one area that is different between Hibernate and OpenJPA >>>>>> -- >>>>>> enhancement processing. >>>>>> >>>>>> In the Tomcat environment, you may be falling back to the subclassing >>>>>> support (which we do not recommend) and hitting a memory leak with >>>>>> that. >>>>>> >>>>>> You said OpenJPA 1.2.x, right? >>>>>> >>>>>> Just a couple of thoughts on the subject... >>>>>> Kevin >>>>>> >>>>>> [1] >>>>>> >>>>>> >>>>>> http://webspherepersistence.blogspot.com/2009/02/openjpa-enhancement.html >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 13, 2010 at 4:25 PM, Kurt T Stam <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> The same code executed straight from a java client (inVM) shows no >>>>>>> memory >>>>>>> leak. >>>>>>> >>>>>>> So is the fact that it is WebService significant then? What else can >>>>>>> be >>>>>>> different? I think one thread remains up, and somehow this causes >>>>>>> openjpa >>>>>>> not being able to clean up after itself. What can I do to debug this >>>>>>> more? I >>>>>>> can actually see in the profiler that the objects are allocated by >>>>>>> the >>>>>>> WebService, but why aren't they cleaned up? >>>>>>> >>>>>>> Thx, >>>>>>> >>>>>>> >>>>>>> --Kurt >>>>>>> >>>>>>> >>>>>>> Kurt T Stam wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Thanks Kevin, thanks for your response. >>>>>>>> >>>>>>>> I just replaced the static call by: >>>>>>>> >>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>> apiAuthToken.setAuthInfo(modelAuthToken.getAuthToken()); >>>>>>>> //MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>> apiAuthToken); >>>>>>>> >>>>>>>> which did not make a difference. >>>>>>>> >>>>>>>> I'm wondering if the fact that my class is a webservice makes a >>>>>>>> difference. I'll try extracting it into >>>>>>>> a regular class with a main method and profile that. At least I know >>>>>>>> that >>>>>>>> I didn't forget something >>>>>>>> completely obvious.. >>>>>>>> >>>>>>>> --Kurt >>>>>>>> >>>>>>>> Kevin Sutter wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Kurt, >>>>>>>>> I agree that this is very common usage of the JPA programming >>>>>>>>> model. >>>>>>>>> And, >>>>>>>>> we are not aware of any memory leaks. About the only thing that >>>>>>>>> jumps >>>>>>>>> out >>>>>>>>> at me is the following two lines: >>>>>>>>> >>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>> apiAuthToken); >>>>>>>>> >>>>>>>>> What do these do? Can you comment these out and see if the memory >>>>>>>>> leak >>>>>>>>> still exists? Since you are passing the modelAuthToken into this >>>>>>>>> method, >>>>>>>>> I >>>>>>>>> don't know what it's doing with the reference and could it be >>>>>>>>> holding >>>>>>>>> onto >>>>>>>>> something to prevent the GC from cleaning up? >>>>>>>>> >>>>>>>>> The rest of your example seems very straight forward with creating >>>>>>>>> and >>>>>>>>> persisting objects. >>>>>>>>> >>>>>>>>> Kevin >>>>>>>>> >>>>>>>>> On Wed, Jan 13, 2010 at 2:09 PM, Rick Curtis <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> If you change the 1000 to something like 1000000... does your >>>>>>>>>> application >>>>>>>>>> go >>>>>>>>>> OOM? Are you running in a JSE environment? What is >>>>>>>>>> PersistenceManager? >>>>>>>>>> >>>>>>>>>> On Wed, Jan 13, 2010 at 2:05 PM, Kurt T Stam <[email protected] >>>>>>>>>> > >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> BTW I'm running with the cache off >>>>>>>>>>> >>>>>>>>>>> <property name="openjpa.DataCache" value="false"/> >>>>>>>>>>> >>>>>>>>>>> (that turns it off right?) >>>>>>>>>>> >>>>>>>>>>> --Kurt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Kurt T Stam wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Hi guys, >>>>>>>>>>>> >>>>>>>>>>>> [DESCRIPTION] The code below inserts a 1000 records in the >>>>>>>>>>>> database. >>>>>>>>>>>> >>>>>>>>>>>> for (int i=1; i<1000; i++) { >>>>>>>>>>>> EntityManager em = PersistenceManager.getEntityManager(); >>>>>>>>>>>> EntityTransaction tx = em.getTransaction(); >>>>>>>>>>>> try { >>>>>>>>>>>> tx.begin(); >>>>>>>>>>>> // Generate auth token and store it! >>>>>>>>>>>> String authInfo = AUTH_TOKEN_PREFIX + >>>>>>>>>>>> UUID.randomUUID(); >>>>>>>>>>>> org.apache.juddi.model.AuthToken modelAuthToken = new >>>>>>>>>>>> org.apache.juddi.model.AuthToken(); >>>>>>>>>>>> if (authInfo != null) { >>>>>>>>>>>> modelAuthToken.setAuthToken(authInfo); >>>>>>>>>>>> modelAuthToken.setCreated(new Date()); >>>>>>>>>>>> modelAuthToken.setLastUsed(new Date()); >>>>>>>>>>>> modelAuthToken.setAuthorizedName(publisherId); >>>>>>>>>>>> modelAuthToken.setNumberOfUses(0); >>>>>>>>>>>> modelAuthToken.setTokenState(AUTHTOKEN_ACTIVE); >>>>>>>>>>>> em.persist(modelAuthToken); >>>>>>>>>>>> } >>>>>>>>>>>> apiAuthToken = new org.uddi.api_v3.AuthToken(); >>>>>>>>>>>> MappingModelToApi.mapAuthToken(modelAuthToken, >>>>>>>>>>>> apiAuthToken); >>>>>>>>>>>> tx.commit(); >>>>>>>>>>>> } finally { >>>>>>>>>>>> if (tx.isActive()) { >>>>>>>>>>>> tx.rollback(); >>>>>>>>>>>> } >>>>>>>>>>>> em.clear(); >>>>>>>>>>>> em.close(); >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [ISSUE] >>>>>>>>>>>> After it leaving this code I end up with a 1000 >>>>>>>>>>>> org.apache.juddi.model.AuthToken objects in memory. I've been >>>>>>>>>>>> using >>>>>>>>>>>> the >>>>>>>>>>>> profiler, and these objects cannot be garbage collected. >>>>>>>>>>>> >>>>>>>>>>>> This seems to be pretty the most common use case of using an >>>>>>>>>>>> OR-mapping >>>>>>>>>>>> tool, so I find it hard to believe openjpa has a memory leak >>>>>>>>>>>> here. >>>>>>>>>>>> Does >>>>>>>>>>>> anyone see what I'm doing wrong? Or can someone point me to an >>>>>>>>>>>> example >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> that >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> does not exhibit this behavior? BTW same code using hibernate >>>>>>>>>>> does >>>>>>>>>>> not >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> accumulate these objects. >>>>>>>>>>>> >>>>>>>>>>>> We're using openjpa 1.2.1. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thx, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Kurt >>>>>>>>>>>> >>>>>>>>>>>> Apache jUDDI. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks, >>>>>>>>>> Rick >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > >
