Hi Jon Latest snapshot worked a charm. Old gen went from 500MB to 110MB after deploy completed. Thanks so much
On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <[email protected]> wrote: > Thanks Jon. > > I'm away at the moment but will give it a spin in the next few days and > let you know. Thanks again > > On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, < > [email protected]> wrote: > >> Hi Paul >> >> I've pushed a new snapshot which (hopefully) should address this. The CI >> build is running on it now. Would you mind giving it a try? If it looks >> good, I'll finish rolling the release. Fingers crossed it looks ok for you. >> >> Jon >> >> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore < >> [email protected]> wrote: >> >>> Further update - I think my changes work with respect to this issue, but >>> I do have a bunch of unit test failures which I need to look into. For >>> info, here's the diff: >>> >>> diff --git >>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java >>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java >>> index a6e2bdc4f5..c0d1a8c98f 100644 >>> --- >>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java >>> +++ >>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java >>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService; >>> import org.apache.openejb.classloader.ClassLoaderConfigurer; >>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer; >>> import org.apache.openejb.component.ClassLoaderEnricher; >>> -import org.apache.openejb.config.ConfigurationFactory; >>> -import org.apache.openejb.config.NewLoaderLogic; >>> -import org.apache.openejb.config.QuickJarsTxtParser; >>> -import org.apache.openejb.config.TldScanner; >>> +import org.apache.openejb.config.*; >>> import org.apache.openejb.core.ConnectorReference; >>> import org.apache.openejb.core.CoreContainerSystem; >>> import org.apache.openejb.core.CoreUserTransaction; >>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool >>> implements org.apache.openejb.spi.A >>> >>> final AppContext appContext = new >>> AppContext(appInfo.appId, SystemInstance.get(), classLoader, >>> globalJndiContext, appJndiContext, appInfo.standaloneModule); >>> appContext.getProperties().putAll(appInfo.properties); >>> + >>> + final Set<Object> keys = >>> appContext.getProperties().keySet(); >>> + for (final Object key : keys) { >>> + if (! Module.class.isInstance(key)) { >>> + appContext.getProperties().remove(key); >>> + } >>> + } >>> + >>> appContext.getInjections().addAll(injections); >>> appContext.getBindings().putAll(globalBindings); >>> appContext.getBindings().putAll(appBindings); >>> diff --git >>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java >>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java >>> index feff954d40..aed0fda941 100644 >>> --- >>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java >>> +++ >>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java >>> @@ -384,12 +384,7 @@ public class JndiEncBuilder { >>> reference = new LinkRef(jndiName); >>> >>> } else if (BeanManager.class.equals(type)) { >>> - reference = new LazyObjectReference<BeanManager>(new >>> Callable<BeanManager>() { >>> - @Override >>> - public BeanManager call() throws Exception { >>> - return new >>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl()); >>> - } >>> - }); >>> + reference = new LazyObjectReference<>(new >>> BeanManagerLazyReference()); >>> >>> } else if (UserTransaction.class.equals(type)) { >>> reference = new >>> IntraVmJndiReference("comp/UserTransaction"); >>> @@ -684,4 +679,11 @@ public class JndiEncBuilder { >>> } >>> } >>> } >>> + >>> + public static class BeanManagerLazyReference implements >>> Callable<BeanManager> { >>> + @Override >>> + public BeanManager call() throws Exception { >>> + return new >>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl()); >>> + } >>> + } >>> } >>> diff --git >>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java >>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java >>> index 2cf387e112..285007925f 100644 >>> --- >>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java >>> +++ >>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java >>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader { >>> private final ClassLoader system; >>> private final boolean embedded; >>> private final boolean parentURLClassLoader; >>> + private final StackTraceElement[] created; >>> >>> // 80% of class files are smaller then 6k >>> private final ByteArrayOutputStream bout = new >>> ByteArrayOutputStream(6 * 1024); >>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader { >>> this.system = ClassLoader.getSystemClassLoader(); >>> this.embedded = this.getClass().getClassLoader() == this.system; >>> this.parentURLClassLoader = >>> URLClassLoader.class.isInstance(parent); >>> + this.created = new Throwable().getStackTrace(); >>> } >>> >>> /* >>> >>> In essence, this does 2 things: >>> >>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a >>> static class. This stops that object from having a reference to >>> JndiEncBuilder and everything that has attached to it. >>> 2. Strips off the EjbModule / WebModule from the SuperProperties on >>> AppContext. This is set when deploying through the apps folder and lives >>> forever. It has the TempClassLoader attached, and therefore all the >>> AnnotationFinder stuff. >>> >>> I suspect (2) is too aggressive and is causing the test failures, but >>> will confirm. >>> >>> Jon >>> >>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore < >>> [email protected]> wrote: >>> >>>> Thanks for sending those over. I'm digging through this - I *think* I >>>> have pinned down a specific reference that stops that classloader from >>>> being released. The behaviour for a war/ear deployed in apps/ seems >>>> different to deploying in webapps/ - I'm assuming that's what you're doing. >>>> If you can confirm, that would help. I'll test out a patch with some bigger >>>> .ear files today. >>>> >>>> Thanks >>>> >>>> Jon >>>> >>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore < >>>> [email protected]> wrote: >>>> >>>>> The mailing list seems to be stripping off your images - can you >>>>> resend and include my email address as well? I'd be keen to dig into this >>>>> a >>>>> little more. >>>>> >>>>> Thanks >>>>> >>>>> Jon >>>>> >>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown >>>>> <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I >>>>>> can see one instance in the heap for every war in my ear. + 1. >>>>>> >>>>>> The TempClassLoader however still has 100's of strong references to >>>>>> it. E.g: >>>>>> >>>>>> [image: image.png] >>>>>> >>>>>> [image: image.png] >>>>>> >>>>>> >>>>>> Paul Carter-Brown >>>>>> Director >>>>>> Jini Guru >>>>>> m: +27 (0) 83 442 7179 <+27834427179> >>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and >>>>>> Leslie >>>>>> Johannesburg, South Africa >>>>>> w: jini.guru e: [email protected] >>>>>> >>>>>> Disclaimer: This message and/or attachment(s) may contain >>>>>> privileged, confidential and/or personal information. If you are not the >>>>>> intended recipient you may not disclose or distribute any of >>>>>> the information contained within this message. In such case you must >>>>>> destroy this message and inform the sender of the error. Jini Guru may >>>>>> not >>>>>> accept liability for any errors, omissions, information and viruses >>>>>> contained in the transmission of this message. Any opinions, conclusions >>>>>> and other information contained within this message not related to Jini >>>>>> Guru official business is deemed to be that of the individual only and is >>>>>> not endorsed by Jini Guru. >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg >>>>>> <[email protected]> wrote: >>>>>> >>>>>>> Hi! >>>>>>> >>>>>>> All this is just a workaround imo. >>>>>>> >>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for >>>>>>> scanning. That means after doing all the class scans we only keep the >>>>>>> extracted information but do not keep the Classes in memory. Doesn't >>>>>>> this >>>>>>> work anymore? >>>>>>> >>>>>>> LieGrue, >>>>>>> strub >>>>>>> >>>>>>> >>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown >>>>>>> <[email protected]>: >>>>>>> > >>>>>>> > FYI. Graph of the change in heap size on a prod instance after >>>>>>> this change: >>>>>>> > >>>>>>> > >>>>>>> > Paul Carter-Brown >>>>>>> > Director >>>>>>> > Jini Guru >>>>>>> > m: +27 (0) 83 442 7179 >>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol >>>>>>> and Leslie >>>>>>> > Johannesburg, South Africa >>>>>>> > w: jini.guru e: [email protected] >>>>>>> > >>>>>>> > Disclaimer: This message and/or attachment(s) may contain >>>>>>> privileged, confidential and/or personal information. If you are not the >>>>>>> intended recipient you may not disclose or distribute any of the >>>>>>> information contained within this message. In such case you must destroy >>>>>>> this message and inform the sender of the error. Jini Guru may not >>>>>>> accept >>>>>>> liability for any errors, omissions, information and viruses contained >>>>>>> in >>>>>>> the transmission of this message. Any opinions, conclusions and other >>>>>>> information contained within this message not related to Jini Guru >>>>>>> official >>>>>>> business is deemed to be that of the individual only and is not >>>>>>> endorsed by >>>>>>> Jini Guru. >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown >>>>>>> <[email protected]> wrote: >>>>>>> > Hi Jon, >>>>>>> > >>>>>>> > I took a look at the source and worked out that I could add my own >>>>>>> filters using openejb.additional.exclude. >>>>>>> > >>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM >>>>>>> heap now sits at around 150MB whereas it would never go below 450MB >>>>>>> previously!!! >>>>>>> > >>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and >>>>>>> hence they are not filtered out and the system works 100%. >>>>>>> > >>>>>>> > My sense is that there should be an easier way to specify what >>>>>>> jars and/or packages to process for annotations. I have come across the >>>>>>> following settings but can't really work out what fits where: >>>>>>> > >>>>>>> > openejb.additional.exclude >>>>>>> > openejb.additional.include >>>>>>> > openejb.deployments.classpath.filter.descriptors >>>>>>> > openejb.deployments.classpath.filter.systemapps >>>>>>> > openejb.deployments.classpath.include >>>>>>> > openejb.deployments.classpath.exclude >>>>>>> > openejb.deployments.package.include >>>>>>> > openejb.deployments.package.exclude >>>>>>> > >>>>>>> > >>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they >>>>>>> are common and yet won't need to be processed for annotations I guess? >>>>>>> > com.amazonaws >>>>>>> > com.fasterxml >>>>>>> > com.google >>>>>>> > com.hazelcast >>>>>>> > io.grpc >>>>>>> > io.netty >>>>>>> > org.docx4j >>>>>>> > org.mongodb >>>>>>> > org.rocksdb >>>>>>> > org.asynchttpclient >>>>>>> > org.apache >>>>>>> > org.aspectj >>>>>>> > >>>>>>> > But then maybe some are too broad and I don't really understand >>>>>>> what the annotation index/cache is really needed for at runtime? Can it >>>>>>> not >>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-( >>>>>>> Seems like the cache uses a significant amount of heap when considering >>>>>>> the >>>>>>> drive towards tiny micro services. >>>>>>> > >>>>>>> > Paul Carter-Brown >>>>>>> > Director >>>>>>> > Jini Guru >>>>>>> > m: +27 (0) 83 442 7179 >>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol >>>>>>> and Leslie >>>>>>> > Johannesburg, South Africa >>>>>>> > w: jini.guru e: [email protected] >>>>>>> > >>>>>>> > Disclaimer: This message and/or attachment(s) may contain >>>>>>> privileged, confidential and/or personal information. If you are not the >>>>>>> intended recipient you may not disclose or distribute any of the >>>>>>> information contained within this message. In such case you must destroy >>>>>>> this message and inform the sender of the error. Jini Guru may not >>>>>>> accept >>>>>>> liability for any errors, omissions, information and viruses contained >>>>>>> in >>>>>>> the transmission of this message. Any opinions, conclusions and other >>>>>>> information contained within this message not related to Jini Guru >>>>>>> official >>>>>>> business is deemed to be that of the individual only and is not >>>>>>> endorsed by >>>>>>> Jini Guru. >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown >>>>>>> <[email protected]> wrote: >>>>>>> > Hi Jon, >>>>>>> > >>>>>>> > Unfortunately the snapshot behaves exactly the same way >>>>>>> > >>>>>>> > Paul Carter-Brown >>>>>>> > Director >>>>>>> > Jini Guru >>>>>>> > m: +27 (0) 83 442 7179 >>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol >>>>>>> and Leslie >>>>>>> > Johannesburg, South Africa >>>>>>> > w: jini.guru e: [email protected] >>>>>>> > >>>>>>> > Disclaimer: This message and/or attachment(s) may contain >>>>>>> privileged, confidential and/or personal information. If you are not the >>>>>>> intended recipient you may not disclose or distribute any of the >>>>>>> information contained within this message. In such case you must destroy >>>>>>> this message and inform the sender of the error. Jini Guru may not >>>>>>> accept >>>>>>> liability for any errors, omissions, information and viruses contained >>>>>>> in >>>>>>> the transmission of this message. Any opinions, conclusions and other >>>>>>> information contained within this message not related to Jini Guru >>>>>>> official >>>>>>> business is deemed to be that of the individual only and is not >>>>>>> endorsed by >>>>>>> Jini Guru. >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore < >>>>>>> [email protected]> wrote: >>>>>>> > Hi Paul >>>>>>> > >>>>>>> > Would you mind trying your application with a recent snapshot? >>>>>>> > >>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/ >>>>>>> . >>>>>>> > I recently updated the exclude list. >>>>>>> > >>>>>>> > Many thanks >>>>>>> > >>>>>>> > Jon >>>>>>> > >>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown >>>>>>> > <[email protected]> wrote: >>>>>>> > >>>>>>> > > Hi, >>>>>>> > > >>>>>>> > > I've been trying to lower the memory footprint of an EAR >>>>>>> deployed to TomEE >>>>>>> > > 8.0.0. >>>>>>> > > >>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is >>>>>>> about >>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my >>>>>>> EAR then >>>>>>> > > the old gen is about 6MB. >>>>>>> > > >>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png] >>>>>>> > > >>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib >>>>>>> directory >>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google >>>>>>> cloud >>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100 >>>>>>> actual >>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I >>>>>>> can see that >>>>>>> > > the memory used by the annotation finder is lowered. >>>>>>> > > >>>>>>> > > Is there any way I can tell TomEE that it need not bother >>>>>>> scanning >>>>>>> > > everything in the /lib folder of my EAR for annotations and >>>>>>> fulling up the >>>>>>> > > heap. I already >>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* >>>>>>> to scan >>>>>>> > > only the one jar in /lib which does have EJB's in it and it >>>>>>> seems to obey >>>>>>> > > this property but it doesn't seem to mean that annotation >>>>>>> processing is >>>>>>> > > skipped for all these other jars in /lib >>>>>>> > > >>>>>>> > > Thanks! >>>>>>> > > >>>>>>> > > Paul Carter-Brown >>>>>>> > > Director >>>>>>> > > Jini Guru >>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179> >>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and >>>>>>> Leslie >>>>>>> > > Johannesburg, South Africa >>>>>>> > > w: jini.guru e: [email protected] >>>>>>> > > >>>>>>> > > Disclaimer: This message and/or attachment(s) may contain >>>>>>> > > privileged, confidential and/or personal information. If you are >>>>>>> not the >>>>>>> > > intended recipient you may not disclose or distribute any of >>>>>>> > > the information contained within this message. In such case you >>>>>>> must >>>>>>> > > destroy this message and inform the sender of the error. Jini >>>>>>> Guru may not >>>>>>> > > accept liability for any errors, omissions, information and >>>>>>> viruses >>>>>>> > > contained in the transmission of this message. Any opinions, >>>>>>> conclusions >>>>>>> > > and other information contained within this message not related >>>>>>> to Jini >>>>>>> > > Guru official business is deemed to be that of the individual >>>>>>> only and is >>>>>>> > > not endorsed by Jini Guru. >>>>>>> > > >>>>>>> > > >>>>>>> >>>>>>>
