Kurt, I'll try to look at your maven plugin configuration next week to check if there's any problem. I like to roll a new release of the openjpa-maven-plugin anyway. I'm just not sure if I should still stick with having a specific openjpa version part of the plugin dependencies, or if I should move it to <scope>provided</scope>. This way the users would need to specify the openjpa dependency of the plugin in any way - but that may also make the use less error prone because the user always will use the correct openjpa version. This would also be a benefit for OpenJPA-1 vs OpenJPA2 (because they have different artifactIds).
LieGrue, strub --- On Fri, 1/15/10, Kurt T Stam <[email protected]> wrote: > From: Kurt T Stam <[email protected]> > Subject: Re: memory leak? - simple question > To: [email protected] > Date: Friday, January 15, 2010, 5:15 PM > Thanks so much Kevin, that did the > trick. Some of the issues you > mentioned where introduced by me not copying right from > the > jUDDI project, but the main issue is that the > openjpa-plugin simply does > not run for some reason. Anyway it is no longer leaking > now! I learned a > lot from this exercise. And yes my class jumped from 2 to > 10 k. BTW I'm > pretty sure this used to work.. > > Thanks again! Going to fix it the jUDDI build now :)! > > --Kurt > > Kevin Sutter wrote: > > Hi Kurt, > > I have not run a profiler to verify the memory leak, > but I think I found the > > problem... > > > > Although you thought you were running the PCEnhancer > during build time, it > > was not being performed. When I looked at the > Tomcat log with the original > > jar file, the following message was logged which > indicates you were falling > > back to the "not ready for production" subclassing > support: > > > > 1844 juddiDatabase > INFO [http-8080-1] openjpa.Enhance - > Creating > > subclass for "[class org.apache.juddi.AuthToken]". > This means that your > > application will be less efficient and will > consume more memory than it > > would if you ran the OpenJPA enhancer. Additionally, > lazy loading will not > > be available for one-to-one and many-to-one persistent > attributes in types > > using field access; they will be loaded eagerly > instead. > > > > I made a few changes to your pom.xml and the "core" > module to get the > > enhancement to work and I believe the memory leak has > went away. Like I > > said, I did not run a profiler, but I added a > "finalize" method to your > > AuthToken class and I now see that these entities are > getting GC'd. > > > > One thing I would do is add the following property to > your persistence.xml. > > > > <property name="openjpa.RuntimeUnenhancedClasses" > value="warn"/> > > > > This property will produce the following warning if > you accidentally fall > > into this subclassing support. We turn this off > by default when running > > within WebSphere since we really don't want customers > to accidentally use > > this subclassing. And, we have turned this off > for 1.3.x and trunk due to > > the various problems associated with it. > Unfortunately, the 1.2.x branch is > > kind of stuck with it... > > > > 1688 juddiDatabase > WARN [http-8080-1] openjpa.Enhance - This > > configuration disallows runtime optimization, but the > following listed types > > were not enhanced at build time or at class load time > with a javaagent: > > "[class org.apache.juddi.AuthToken]". > > > > And, of course, your webservice no longer works > either. You will get an > > HTTP Status 500 error with an error message and call > stack that further > > explains the error. > > > > To get the maven plugin to work properly... I > made several changes, so > > hopefully I remember everything... > > > > o The number one thing is that the > openjpa:enhance goal doesn't seem to get > > automatically invoked when you compile. I have > no idea why, but that > > doesn't seem to work. > > > > o You have to make your persistence.xml file > available to the "core" > > module. For my testing, I just created a new > directory in the "core" module > > called > "\openjpa-leak\core\src\main\resources\META-INF\persistence.xml". > > This will automatically get copied over to "target" > and then the > > openjpa:enhance goal will find it. > > > > o How you want to make your persistence.xml file > available to both your > > "core" module and the "leak-war" module is up to > you... :-) > > > > o I added a dependency in your core\pom.xml file > for the 1.2.1 openjpa. > > The documentation [1] says it will default to 1.2.0, > if none is specified. > > > > > <dependencies> > > > <dependency> > > > > <groupId>org.apache.openjpa</groupId> > > > > <artifactId>openjpa</artifactId> > > > > <version>1.2.1</version> > > > </dependency> > > > </dependencies> > > > > o Your plugin properties don't seem to be > specified correctly, at least > > according to the documentation, but I didn't touch > them and things still > > seemed to work (once I got around the other > problems). > > > > o I also changed where to find the entities to > enhance since this pom.xml > > still specified the "model" directory. There > also seems to be confusion as > > to whether to specify <classes> or > <includes>. I went with <includes> since > > that's what the example showed. > > > > o It's very easy to see if the enhancement > process worked or not since the > > size of your entity classes increases by a few > thousand bytes. And, with > > the extra property above, this will prevent you from > accidentally using the > > unenhanced version. > > > > Hope this helps! Let me know if this gets you > around the memory leak. > > > > Kevin > > > > [1] http://mojo.codehaus.org/openjpa-maven-plugin/usage.html > > > > > > > > On Thu, Jan 14, 2010 at 2:12 PM, Kurt T Stam <[email protected]> > wrote: > > > > > >> 1. Cool that you got it running :) > >> > >> 2. I just ran with openjpa 1.2.2.SNAPSHOT and > still see an accumulation of > >> AuthToken objects. > >> > >> --K > >> > >> > >> Kevin Sutter wrote: > >> > >> > >>> Hey, this wasn't so difficult... > :-) Maybe I can still learn new > >>> things... I have Tomcat up and running > and I can get your app running. > >>> > >>> On Thu, Jan 14, 2010 at 1:46 PM, Kevin Sutter > <[email protected]> > wrote: > >>> > >>> > >>> > >>> > >>>> Hi Kurt, > >>>> These instructions, of course, assume that > I know something about Tomcat. > >>>> This would not be a good > assumption... :-) If you really want me to see > >>>> this in action, I will need more > information on how to start Tomcat and > >>>> how > >>>> to load applications. In the mean > time, a trace log of OpenJPA would be > >>>> an > >>>> excellent start. > >>>> > >>>> Thanks! > >>>> Kevin > >>>> > >>>> > >>>> On Thu, Jan 14, 2010 at 11:30 AM, Kurt T > Stam <[email protected]> > >>>> wrote: > >>>> > >>>> > >>>> > >>>> > >>>>> Hi Kevin, > >>>>> > >>>>> I tried adding the javaagent stuff, > and it still ends up leaking. I > >>>>> extracted the code into a tiny > >>>>> war with just one servlet which > persist the authToken. The build creates > >>>>> a > >>>>> Tomcat distro backed by a > >>>>> derby database. You can download it > from: > >>>>> > >>>>> http://people.apache.org/~kstam/leak.tgz<http://people.apache.org/%7Ekstam/leak.tgz> > >>>>> <http://people.apache.org/%7Ekstam/leak.tgz> > >>>>> > >>>>> > >>>>> You can build it with: mvn install, > you can find the final tomcat in > >>>>> tomcat/target/tomcat-1.0.SNAPSHOT.zip > >>>>> > >>>>> or if you don't want to build it: > >>>>> http://people.apache.org/~kstam/tomcat-1.0.SNAPSHOT.zip<http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> > >>>>> <http://people.apache.org/%7Ekstam/tomcat-1.0.SNAPSHOT.zip> > >>>>> > >>>>> > >>>>> After you bring up tomcat it > persistent an authToken when you hit > >>>>> http://localhost:8080/leak/test > >>>>> > >>>>> In the profiler I see an > org.apache.juddi.AuthToken left behind for each > >>>>> request. > >>>>> > >>>>> Hopefully this little example will > pinpoint what's going on. > >>>>> > >>>>> Thx Kev for sticking with me :) > >>>>> > >>>>> --Kurt > >>>>> > >>>>> > >>>>> > >>>>> Kevin Sutter wrote: > >>>>> > >>>>> > >>>>> > >>>>> > > >>>>>> Hi Kurt, > >>>>>> Since you have the environment to > reproduce this, could you run with > >>>>>> the > >>>>>> OpenJPA trace enabled? I'm > not even sure what I'm looking for at this > >>>>>> point. :-) But, maybe > there's some tracing that would point us in the > >>>>>> right direction. > >>>>>> > >>>>>> <property name="openjpa.Log" > value="DefaultLevel=TRACE"/> > >>>>>> > >>>>>> We have been always been on the > watch for memory leaks, so this one is > >>>>>> puzzling. You did mention > blocked threads as well. At this point, you > >>>>>> are > >>>>>> probably hitting blocked threads > due to the GC and Memory problems. > >>>>>> But, > >>>>>> we > >>>>>> did recently resolve some locking > situations [1] in our > >>>>>> MetaDataRepository > >>>>>> that could affect scaling (under > extreme load). This was resolved in > >>>>>> the > >>>>>> 1.2.x SNAPSHOT drivers (Mike is > currently pushing for a 1.2.2 release, > >>>>>> so > >>>>>> that should be coming soon). > >>>>>> > >>>>>> Does your WebService version of > the problem use EntityManager > >>>>>> transactions, > >>>>>> or is the transaction service > coming from Tomcat? And, does your > >>>>>> WebService > >>>>>> version also do the em.clear() and > em.close()? Basically, are you > >>>>>> using > >>>>>> application-managed or > container-managed persistence while running as a > >>>>>> WebService? > >>>>>> > >>>>>> I'm very interested in helping to > resolve this issue. Like I > >>>>>> mentioned, > >>>>>> we > >>>>>> do a lot of memory leakage and > scalability testing of the OpenJPA > >>>>>> solution > >>>>>> and we're not aware of > anything. Although there is always the > >>>>>> possibility > >>>>>> that something was resolved in the > 1.2.x branch after the 1.2.1 release > >>>>>> that > >>>>>> I'm not remembering... > >>>>>> > >>>>>> Thanks for your help, > >>>>>> Kevin > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/OPENJPA-250 > >>>>>> > >>>>>> On Wed, Jan 13, 2010 at 9:27 PM, > Kurt T Stam <[email protected]> > >>>>>> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > > >>>>>>> Thanks Kevin, > >>>>>>> > >>>>>>> We're enhancing at build > time: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> http://svn.apache.org/repos/asf/webservices/juddi/trunk/juddi-core/pom.xml > >>>>>>> > >>>>>>> Yeah we've been running load > tests and things are nice and stable with > >>>>>>> Hibernate but with Openjpa we > see increasing memory use, blocking > >>>>>>> threads > >>>>>>> and then an OOM. http://issues.apache.org/jira/browse/JUDDI-267. Our > >>>>>>> preference would be to ship > with openjpa by default; but our build > >>>>>>> supports > >>>>>>> both hibernate and openjpa. > >>>>>>> > >>>>>>> And yes we use openjpa 1.2.1 > (latest stable version). > >>>>>>> > >>>>>>> --Kurt > >>>>>>> > >>>>>>> > >>>>>>> Kevin Sutter wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > > >>>>>>>> Interesting detective > work, Kurt. Thanks. > >>>>>>>> > >>>>>>>> Why the WebService version > of the app would behave differently as far > >>>>>>>> as > >>>>>>>> GC > >>>>>>>> is concerned is a > mystery. And, you said that plugging in Hibernate > >>>>>>>> into > >>>>>>>> this scenario, everything > works okay? Very confusing. > >>>>>>>> > >>>>>>>> How are you performing the > Entity enhancement processing? Are you > >>>>>>>> pre-enhancing via your > build process? Or, are you using the > >>>>>>>> -javaagent > >>>>>>>> mechanism? Or, are > you falling back to the subclassing support > >>>>>>>> within > >>>>>>>> OpenJPA? (See [1] > for more information on these questions in case > >>>>>>>> they > >>>>>>>> don't make sense.) > >>>>>>>> > >>>>>>>> This would be one area > that is different between Hibernate and > >>>>>>>> OpenJPA > >>>>>>>> -- > >>>>>>>> enhancement processing. > >>>>>>>> > >>>>>>>> In the Tomcat environment, > you may be falling back to the subclassing > >>>>>>>> support (which we do not > recommend) and hitting a memory leak with > >>>>>>>> that. > >>>>>>>> > >>>>>>>> You said OpenJPA 1.2.x, > right? > >>>>>>>> > >>>>>>>> Just a couple of thoughts > on the subject... > >>>>>>>> Kevin > >>>>>>>> > >>>>>>>> [1] > >>>>>>>> > >>>>>>>> > >>>>>>>> http://webspherepersistence.blogspot.com/2009/02/openjpa-enhancement.html > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, Jan 13, 2010 at > 4:25 PM, Kurt T Stam <[email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > > >>>>>>>>> The same code executed > straight from a java client (inVM) shows no > >>>>>>>>> memory > >>>>>>>>> leak. > >>>>>>>>> > >>>>>>>>> So is the fact that it > is WebService significant then? What else can > >>>>>>>>> be > >>>>>>>>> different? I think one > thread remains up, and somehow this causes > >>>>>>>>> openjpa > >>>>>>>>> not being able to > clean up after itself. What can I do to debug this > >>>>>>>>> more? I > >>>>>>>>> can actually see in > the profiler that the objects are allocated by > >>>>>>>>> the > >>>>>>>>> WebService, but why > aren't they cleaned up? > >>>>>>>>> > >>>>>>>>> Thx, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> --Kurt > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Kurt T Stam wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > > >>>>>>>>>> Thanks Kevin, > thanks for your response. > >>>>>>>>>> > >>>>>>>>>> I just replaced > the static call by: > >>>>>>>>>> > >>>>>>>>>> > apiAuthToken = new > org.uddi.api_v3.AuthToken(); > >>>>>>>>>> > > apiAuthToken.setAuthInfo(modelAuthToken.getAuthToken()); > >>>>>>>>>> > > //MappingModelToApi.mapAuthToken(modelAuthToken, > >>>>>>>>>> apiAuthToken); > >>>>>>>>>> > >>>>>>>>>> which did not make > a difference. > >>>>>>>>>> > >>>>>>>>>> I'm wondering if > the fact that my class is a webservice makes a > >>>>>>>>>> difference. I'll > try extracting it into > >>>>>>>>>> a regular class > with a main method and profile that. At least I > >>>>>>>>>> know > >>>>>>>>>> that > >>>>>>>>>> I didn't forget > something > >>>>>>>>>> completely > obvious.. > >>>>>>>>>> > >>>>>>>>>> --Kurt > >>>>>>>>>> > >>>>>>>>>> Kevin Sutter > wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > > > >>>>>>>>>>> Kurt, > >>>>>>>>>>> I agree that > this is very common usage of the JPA programming > >>>>>>>>>>> model. > >>>>>>>>>>> And, > >>>>>>>>>>> we are not > aware of any memory leaks. About the only thing that > >>>>>>>>>>> jumps > >>>>>>>>>>> out > >>>>>>>>>>> at me is the > following two lines: > >>>>>>>>>>> > >>>>>>>>>>> > apiAuthToken = new > org.uddi.api_v3.AuthToken(); > >>>>>>>>>>> > > MappingModelToApi.mapAuthToken(modelAuthToken, > >>>>>>>>>>> > apiAuthToken); > >>>>>>>>>>> > >>>>>>>>>>> What do these > do? Can you comment these out and see if the memory > >>>>>>>>>>> leak > >>>>>>>>>>> still > exists? Since you are passing the modelAuthToken into > this > >>>>>>>>>>> method, > >>>>>>>>>>> I > >>>>>>>>>>> don't know > what it's doing with the reference and could it be > >>>>>>>>>>> holding > >>>>>>>>>>> onto > >>>>>>>>>>> something to > prevent the GC from cleaning up? > >>>>>>>>>>> > >>>>>>>>>>> The rest of > your example seems very straight forward with creating > >>>>>>>>>>> and > >>>>>>>>>>> persisting > objects. > >>>>>>>>>>> > >>>>>>>>>>> Kevin > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Jan > 13, 2010 at 2:09 PM, Rick Curtis <[email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > > > >>>>>>>>>>>> If you > change the 1000 to something like 1000000... does your > >>>>>>>>>>>> > application > >>>>>>>>>>>> go > >>>>>>>>>>>> OOM? Are > you running in a JSE environment? What is > >>>>>>>>>>>> > PersistenceManager? > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, > Jan 13, 2010 at 2:05 PM, Kurt T Stam < > >>>>>>>>>>>> [email protected]> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > > > >>>>>>>>>>>>> BTW > I'm running with the cache off > >>>>>>>>>>>>> > >>>>>>>>>>>>> > <property name="openjpa.DataCache" value="false"/> > >>>>>>>>>>>>> > >>>>>>>>>>>>> (that > turns it off right?) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > --Kurt > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Kurt T > Stam wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi > guys, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > [DESCRIPTION] The code below inserts a 1000 records in the > >>>>>>>>>>>>>> > database. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > for (int i=1; i<1000; i++) { > >>>>>>>>>>>>>> > EntityManager em = > PersistenceManager.getEntityManager(); > >>>>>>>>>>>>>> > EntityTransaction tx = > em.getTransaction(); > >>>>>>>>>>>>>> > try { > >>>>>>>>>>>>>> > tx.begin(); > >>>>>>>>>>>>>> > // > Generate auth token and store it! > >>>>>>>>>>>>>> > String authInfo = > AUTH_TOKEN_PREFIX + > >>>>>>>>>>>>>> > UUID.randomUUID(); > >>>>>>>>>>>>>> > > org.apache.juddi.model.AuthToken > modelAuthToken = new > >>>>>>>>>>>>>> > org.apache.juddi.model.AuthToken(); > >>>>>>>>>>>>>> > if (authInfo != null) > { > >>>>>>>>>>>>>> > > modelAuthToken.setAuthToken(authInfo); > >>>>>>>>>>>>>> > > modelAuthToken.setCreated(new Date()); > >>>>>>>>>>>>>> > > modelAuthToken.setLastUsed(new Date()); > >>>>>>>>>>>>>> > > modelAuthToken.setAuthorizedName(publisherId); > >>>>>>>>>>>>>> > > modelAuthToken.setNumberOfUses(0); > >>>>>>>>>>>>>> > > modelAuthToken.setTokenState(AUTHTOKEN_ACTIVE); > >>>>>>>>>>>>>> > > em.persist(modelAuthToken); > >>>>>>>>>>>>>> > } > >>>>>>>>>>>>>> > apiAuthToken = > new org.uddi.api_v3.AuthToken(); > >>>>>>>>>>>>>> > > MappingModelToApi.mapAuthToken(modelAuthToken, > >>>>>>>>>>>>>> > apiAuthToken); > >>>>>>>>>>>>>> > tx.commit(); > >>>>>>>>>>>>>> > > } finally { > >>>>>>>>>>>>>> > if (tx.isActive()) { > >>>>>>>>>>>>>> > > tx.rollback(); > >>>>>>>>>>>>>> > } > >>>>>>>>>>>>>> > em.clear(); > >>>>>>>>>>>>>> > em.close(); > >>>>>>>>>>>>>> > } > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > [ISSUE] > >>>>>>>>>>>>>> > After it leaving this code I end up with a 1000 > >>>>>>>>>>>>>> > org.apache.juddi.model.AuthToken objects in memory. I've > been > >>>>>>>>>>>>>> > using > >>>>>>>>>>>>>> > the > >>>>>>>>>>>>>> > profiler, and these objects cannot be garbage collected. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > This seems to be pretty the most common use case of using > an > >>>>>>>>>>>>>> > OR-mapping > >>>>>>>>>>>>>> > tool, so I find it hard to believe openjpa has a memory > leak > >>>>>>>>>>>>>> > here. > >>>>>>>>>>>>>> > Does > >>>>>>>>>>>>>> > anyone see what I'm doing wrong? Or can someone point me to > an > >>>>>>>>>>>>>> > example > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> that > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> does > not exhibit this behavior? BTW same code using hibernate > >>>>>>>>>>>>> does > >>>>>>>>>>>>> not > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > accumulate these objects. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > We're using openjpa 1.2.1. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thx, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Kurt > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Apache jUDDI. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> -- > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Rick > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > > > >>>>>>>>>>> > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > >>>>>> > > >>>>> > > >>> > >> > > > > > >
