Thanks Jon.
I'm away at the moment but will give it a spin in the next few days and
let you know. Thanks again
On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <jonathan.gallim...@gmail.com>
wrote:
> Hi Paul
>
> I've pushed a new snapshot which (hopefully) should address this. The CI
> build is running on it now. Would you mind giving it a try? If it looks
> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>
> Jon
>
> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
> jonathan.gallim...@gmail.com> wrote:
>
>> Further update - I think my changes work with respect to this issue, but
>> I do have a bunch of unit test failures which I need to look into. For
>> info, here's the diff:
>>
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> index a6e2bdc4f5..c0d1a8c98f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>> import org.apache.openejb.classloader.ClassLoaderConfigurer;
>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>> import org.apache.openejb.component.ClassLoaderEnricher;
>> -import org.apache.openejb.config.ConfigurationFactory;
>> -import org.apache.openejb.config.NewLoaderLogic;
>> -import org.apache.openejb.config.QuickJarsTxtParser;
>> -import org.apache.openejb.config.TldScanner;
>> +import org.apache.openejb.config.*;
>> import org.apache.openejb.core.ConnectorReference;
>> import org.apache.openejb.core.CoreContainerSystem;
>> import org.apache.openejb.core.CoreUserTransaction;
>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>> implements org.apache.openejb.spi.A
>>
>> final AppContext appContext = new
>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>> appContext.getProperties().putAll(appInfo.properties);
>> +
>> + final Set<Object> keys =
>> appContext.getProperties().keySet();
>> + for (final Object key : keys) {
>> + if (! Module.class.isInstance(key)) {
>> + appContext.getProperties().remove(key);
>> + }
>> + }
>> +
>> appContext.getInjections().addAll(injections);
>> appContext.getBindings().putAll(globalBindings);
>> appContext.getBindings().putAll(appBindings);
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> index feff954d40..aed0fda941 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>> reference = new LinkRef(jndiName);
>>
>> } else if (BeanManager.class.equals(type)) {
>> - reference = new LazyObjectReference<BeanManager>(new
>> Callable<BeanManager>() {
>> - @Override
>> - public BeanManager call() throws Exception {
>> - return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> - }
>> - });
>> + reference = new LazyObjectReference<>(new
>> BeanManagerLazyReference());
>>
>> } else if (UserTransaction.class.equals(type)) {
>> reference = new
>> IntraVmJndiReference("comp/UserTransaction");
>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>> }
>> }
>> }
>> +
>> + public static class BeanManagerLazyReference implements
>> Callable<BeanManager> {
>> + @Override
>> + public BeanManager call() throws Exception {
>> + return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> + }
>> + }
>> }
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> index 2cf387e112..285007925f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>> private final ClassLoader system;
>> private final boolean embedded;
>> private final boolean parentURLClassLoader;
>> + private final StackTraceElement[] created;
>>
>> // 80% of class files are smaller then 6k
>> private final ByteArrayOutputStream bout = new
>> ByteArrayOutputStream(6 * 1024);
>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>> this.system = ClassLoader.getSystemClassLoader();
>> this.embedded = this.getClass().getClassLoader() == this.system;
>> this.parentURLClassLoader =
>> URLClassLoader.class.isInstance(parent);
>> + this.created = new Throwable().getStackTrace();
>> }
>>
>> /*
>>
>> In essence, this does 2 things:
>>
>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
>> static class. This stops that object from having a reference to
>> JndiEncBuilder and everything that has attached to it.
>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>> AppContext. This is set when deploying through the apps folder and lives
>> forever. It has the TempClassLoader attached, and therefore all the
>> AnnotationFinder stuff.
>>
>> I suspect (2) is too aggressive and is causing the test failures, but
>> will confirm.
>>
>> Jon
>>
>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>> jonathan.gallim...@gmail.com> wrote:
>>
>>> Thanks for sending those over. I'm digging through this - I *think* I
>>> have pinned down a specific reference that stops that classloader from
>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>> .ear files today.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>> jonathan.gallim...@gmail.com> wrote:
>>>
>>>> The mailing list seems to be stripping off your images - can you resend
>>>> and include my email address as well? I'd be keen to dig into this a little
>>>> more.
>>>>
>>>> Thanks
>>>>
>>>> Jon
>>>>
>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>> <paul.carter-br...@jini.guru> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>
>>>>> The TempClassLoader however still has 100's of strong references to
>>>>> it. E.g:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>> Paul Carter-Brown
>>>>> Director
>>>>> Jini Guru
>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> Johannesburg, South Africa
>>>>> w: jini.guru e: p...@jini.guru
>>>>>
>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of
>>>>> the information contained within this message. In such case you must
>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>> accept liability for any errors, omissions, information and viruses
>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>> and other information contained within this message not related to Jini
>>>>> Guru official business is deemed to be that of the individual only and is
>>>>> not endorsed by Jini Guru.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>> <strub...@yahoo.de.invalid> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> All this is just a workaround imo.
>>>>>>
>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>> work anymore?
>>>>>>
>>>>>> LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>> <paul.carter-br...@jini.guru>:
>>>>>> >
>>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>>> change:
>>>>>> >
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: p...@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru
>>>>>> official
>>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>>> by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>> <paul.carter-br...@jini.guru> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > I took a look at the source and worked out that I could add my own
>>>>>> filters using openejb.additional.exclude.
>>>>>> >
>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>> previously!!!
>>>>>> >
>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>>> they are not filtered out and the system works 100%.
>>>>>> >
>>>>>> > My sense is that there should be an easier way to specify what jars
>>>>>> and/or packages to process for annotations. I have come across the
>>>>>> following settings but can't really work out what fits where:
>>>>>> >
>>>>>> > openejb.additional.exclude
>>>>>> > openejb.additional.include
>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>> > openejb.deployments.classpath.include
>>>>>> > openejb.deployments.classpath.exclude
>>>>>> > openejb.deployments.package.include
>>>>>> > openejb.deployments.package.exclude
>>>>>> >
>>>>>> >
>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>>> > com.amazonaws
>>>>>> > com.fasterxml
>>>>>> > com.google
>>>>>> > com.hazelcast
>>>>>> > io.grpc
>>>>>> > io.netty
>>>>>> > org.docx4j
>>>>>> > org.mongodb
>>>>>> > org.rocksdb
>>>>>> > org.asynchttpclient
>>>>>> > org.apache
>>>>>> > org.aspectj
>>>>>> >
>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>> what the annotation index/cache is really needed for at runtime? Can it
>>>>>> not
>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>> Seems like the cache uses a significant amount of heap when considering
>>>>>> the
>>>>>> drive towards tiny micro services.
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: p...@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru
>>>>>> official
>>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>>> by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>> <paul.carter-br...@jini.guru> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: p...@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru
>>>>>> official
>>>>>> business is deemed to be that of the individual only and is not endorsed
>>>>>> by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>> jonathan.gallim...@gmail.com> wrote:
>>>>>> > Hi Paul
>>>>>> >
>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>> .
>>>>>> > I recently updated the exclude list.
>>>>>> >
>>>>>> > Many thanks
>>>>>> >
>>>>>> > Jon
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>> > <paul.carter-br...@jini.guru> wrote:
>>>>>> >
>>>>>> > > Hi,
>>>>>> > >
>>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>>> to TomEE
>>>>>> > > 8.0.0.
>>>>>> > >
>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>>> about
>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>> EAR then
>>>>>> > > the old gen is about 6MB.
>>>>>> > >
>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>> > >
>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>> directory
>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>>> cloud
>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>> actual
>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>> can see that
>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>> > >
>>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>> fulling up the
>>>>>> > > heap. I already
>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>> to scan
>>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>>> to obey
>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>> processing is
>>>>>> > > skipped for all these other jars in /lib
>>>>>> > >
>>>>>> > > Thanks!
>>>>>> > >
>>>>>> > > Paul Carter-Brown
>>>>>> > > Director
>>>>>> > > Jini Guru
>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > > Johannesburg, South Africa
>>>>>> > > w: jini.guru e: p...@jini.guru
>>>>>> > >
>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>> > > privileged, confidential and/or personal information. If you are
>>>>>> not the
>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>> > > the information contained within this message. In such case you
>>>>>> must
>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>> Guru may not
>>>>>> > > accept liability for any errors, omissions, information and
>>>>>> viruses
>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>> conclusions
>>>>>> > > and other information contained within this message not related
>>>>>> to Jini
>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>> only and is
>>>>>> > > not endorsed by Jini Guru.
>>>>>> > >
>>>>>> > >
>>>>>>
>>>>>>
