Hi Paul
I've pushed a new snapshot which (hopefully) should address this. The CI
build is running on it now. Would you mind giving it a try? If it looks
good, I'll finish rolling the release. Fingers crossed it looks ok for you.
Jon
On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
jonathan.gallim...@gmail.com> wrote:
> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
> import org.apache.openejb.classloader.ClassLoaderConfigurer;
> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
> import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
> import org.apache.openejb.core.ConnectorReference;
> import org.apache.openejb.core.CoreContainerSystem;
> import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
> final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
> appContext.getProperties().putAll(appInfo.properties);
> +
> + final Set<Object> keys =
> appContext.getProperties().keySet();
> + for (final Object key : keys) {
> + if (! Module.class.isInstance(key)) {
> + appContext.getProperties().remove(key);
> + }
> + }
> +
> appContext.getInjections().addAll(injections);
> appContext.getBindings().putAll(globalBindings);
> appContext.getBindings().putAll(appBindings);
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
> reference = new LinkRef(jndiName);
>
> } else if (BeanManager.class.equals(type)) {
> - reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> - @Override
> - public BeanManager call() throws Exception {
> - return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> - }
> - });
> + reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
> } else if (UserTransaction.class.equals(type)) {
> reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
> }
> }
> }
> +
> + public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> + @Override
> + public BeanManager call() throws Exception {
> + return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> + }
> + }
> }
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
> private final ClassLoader system;
> private final boolean embedded;
> private final boolean parentURLClassLoader;
> + private final StackTraceElement[] created;
>
> // 80% of class files are smaller then 6k
> private final ByteArrayOutputStream bout = new
> ByteArrayOutputStream(6 * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
> this.system = ClassLoader.getSystemClassLoader();
> this.embedded = this.getClass().getClassLoader() == this.system;
> this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> + this.created = new Throwable().getStackTrace();
> }
>
> /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> jonathan.gallim...@gmail.com> wrote:
>
>> Thanks for sending those over. I'm digging through this - I *think* I
>> have pinned down a specific reference that stops that classloader from
>> being released. The behaviour for a war/ear deployed in apps/ seems
>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>> If you can confirm, that would help. I'll test out a patch with some bigger
>> .ear files today.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>> jonathan.gallim...@gmail.com> wrote:
>>
>>> The mailing list seems to be stripping off your images - can you resend
>>> and include my email address as well? I'd be keen to dig into this a little
>>> more.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>> <paul.carter-br...@jini.guru> wrote:
>>>
>>>> Hi,
>>>>
>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>
>>>> The TempClassLoader however still has 100's of strong references to it.
>>>> E.g:
>>>>
>>>> [image: image.png]
>>>>
>>>> [image: image.png]
>>>>
>>>>
>>>> Paul Carter-Brown
>>>> Director
>>>> Jini Guru
>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>>> Johannesburg, South Africa
>>>> w: jini.guru e: p...@jini.guru
>>>>
>>>> Disclaimer: This message and/or attachment(s) may contain
>>>> privileged, confidential and/or personal information. If you are not the
>>>> intended recipient you may not disclose or distribute any of
>>>> the information contained within this message. In such case you must
>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>> accept liability for any errors, omissions, information and viruses
>>>> contained in the transmission of this message. Any opinions, conclusions
>>>> and other information contained within this message not related to Jini
>>>> Guru official business is deemed to be that of the individual only and is
>>>> not endorsed by Jini Guru.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <strub...@yahoo.de.invalid>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> All this is just a workaround imo.
>>>>>
>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>> scanning. That means after doing all the class scans we only keep the
>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>> work anymore?
>>>>>
>>>>> LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>> <paul.carter-br...@jini.guru>:
>>>>> >
>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>> change:
>>>>> >
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: p...@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru
>>>>> official
>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>> by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>> <paul.carter-br...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > I took a look at the source and worked out that I could add my own
>>>>> filters using openejb.additional.exclude.
>>>>> >
>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>> previously!!!
>>>>> >
>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>> they are not filtered out and the system works 100%.
>>>>> >
>>>>> > My sense is that there should be an easier way to specify what jars
>>>>> and/or packages to process for annotations. I have come across the
>>>>> following settings but can't really work out what fits where:
>>>>> >
>>>>> > openejb.additional.exclude
>>>>> > openejb.additional.include
>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>> > openejb.deployments.classpath.include
>>>>> > openejb.deployments.classpath.exclude
>>>>> > openejb.deployments.package.include
>>>>> > openejb.deployments.package.exclude
>>>>> >
>>>>> >
>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>> > com.amazonaws
>>>>> > com.fasterxml
>>>>> > com.google
>>>>> > com.hazelcast
>>>>> > io.grpc
>>>>> > io.netty
>>>>> > org.docx4j
>>>>> > org.mongodb
>>>>> > org.rocksdb
>>>>> > org.asynchttpclient
>>>>> > org.apache
>>>>> > org.aspectj
>>>>> >
>>>>> > But then maybe some are too broad and I don't really understand what
>>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>>> like the cache uses a significant amount of heap when considering the
>>>>> drive
>>>>> towards tiny micro services.
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: p...@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru
>>>>> official
>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>> by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>> <paul.carter-br...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: p...@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru
>>>>> official
>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>> by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>> jonathan.gallim...@gmail.com> wrote:
>>>>> > Hi Paul
>>>>> >
>>>>> > Would you mind trying your application with a recent snapshot?
>>>>> >
>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>> .
>>>>> > I recently updated the exclude list.
>>>>> >
>>>>> > Many thanks
>>>>> >
>>>>> > Jon
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>> > <paul.carter-br...@jini.guru> wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>> to TomEE
>>>>> > > 8.0.0.
>>>>> > >
>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>> about
>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>> EAR then
>>>>> > > the old gen is about 6MB.
>>>>> > >
>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>> > >
>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>> directory
>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>> cloud
>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>> actual
>>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>>> see that
>>>>> > > the memory used by the annotation finder is lowered.
>>>>> > >
>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>> fulling up the
>>>>> > > heap. I already
>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>>> scan
>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>> to obey
>>>>> > > this property but it doesn't seem to mean that annotation
>>>>> processing is
>>>>> > > skipped for all these other jars in /lib
>>>>> > >
>>>>> > > Thanks!
>>>>> > >
>>>>> > > Paul Carter-Brown
>>>>> > > Director
>>>>> > > Jini Guru
>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > > Johannesburg, South Africa
>>>>> > > w: jini.guru e: p...@jini.guru
>>>>> > >
>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>> > > privileged, confidential and/or personal information. If you are
>>>>> not the
>>>>> > > intended recipient you may not disclose or distribute any of
>>>>> > > the information contained within this message. In such case you
>>>>> must
>>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>>> may not
>>>>> > > accept liability for any errors, omissions, information and viruses
>>>>> > > contained in the transmission of this message. Any opinions,
>>>>> conclusions
>>>>> > > and other information contained within this message not related to
>>>>> Jini
>>>>> > > Guru official business is deemed to be that of the individual only
>>>>> and is
>>>>> > > not endorsed by Jini Guru.
>>>>> > >
>>>>> > >
>>>>>
>>>>>
