I was finally able to take some time and find out where multiple threads could be showing up in the environment where my application was not doing any multithreaded work with the Drools knowledge base or session.
One example I have can be seen by looking at the `org.drools.rule.constraint.MvelConstraint` class. This method `#jitEvaluator(Object, InternalWorkingMemory, LeftTuple)` has a line: `ExecutorHolder.executor.execute(new ConditionJitter(this, object, workingMemory, leftTuple));` This is roughly around this line in 5.5.0.Final, for example. https://github.com/droolsjbpm/drools/blob/5.5.0.Final/drools-core/src/main/java/org/drools/rule/constraint/MvelConstraint.java#L232 My understanding of this method functionality is that a `java.util.concurrent.Executor` is used to execute a Runnable. The Executor returned is configurable via the `org.drools.util.ServiceRegistry`. There is a related Drools RuleBaseConfiguration option called "executorService" that allows the user to configure the details of how this Runnable is executed. e.g. a custom application thread pool, a simple thread, no new thread at all - synchronously. Drools has a default for this executorService, which is @ https://github.com/droolsjbpm/drools/blob/5.5.0.Final/drools-core/src/main/java/org/drools/concurrent/ExecutorProviderImpl.java#L11 This implementation runs the Runnable in a new thread, this is where a concurrency issue is able to come into the picture. So in the MvelConstraint#jitEvaluator example I mention above, uses this executor to perform some of the Jit compilation on MVEL constraints in a new thread, to not "burden" the current thread with this work. The issue is that the Runnable work that is done on another thread by the ConditionJitter, ultimately works with the same `org.drools.util.CompositeClassLoader$CachingLoader` as the main/other thread(s). This CachingLoader is not safe for concurrent access to its `classLoaderResultMap` field, since it is implemented as a `j.u.HashMap`. e.g. if one thread is reading from the cache at the same time as another thread is doing a #put into the cache and the cache resizes, this can lead to the reading going into a non-terminating loop. So, in summary, my application has no multithreading being done with regard to the main thread working with the Drools knowledge base and session. However, when some of the knowledge base is being compiled/Jitted there are multiple threads doing work, and on some occasions they we run into a concurrent access to the CachingLoader cache problem resulting in a non-terminating loop. I believe this would fall into the category of a "race condition" in that I've only observed the issue happening about 1-5% of the time of the same knowledge base being built. I'll am being general on saying "built" here, as I am not sure this is the correct terminology to describe the time that MvelConstraint's are being/attempted to be Jitted and this is happening. I'll note that "turning off" jitting didn't fix the problem. There is a chance that I did not effectively stop all jitting though. I attempted to turn it off, by what is explained @ http://mvel.codehaus.org/Disabling+the+JIT+Compiler. I cannot remember if I've tried to turn it off via "drools.permgenThreshold" in my experiments with this issue though. (That one is described @ http://drools.46999.n3.nabble.com/rules-users-Out-of-Mem-and-thousand-threads-Disabling-JIT-td4026694.html) I noticed that with jitting "turned off" this method was still being called when a MvelConstraint was found to be "dynamic" via the `isDynamic` boolean flag of the MvelConstraint class. I'm not clear on what causes a constraint to be considered dynamic or not. The workaround that *did* work for me was to just disable the CacheLoader via the configuration property "drools.classLoaderCacheEnabled". I do not believe I've seen significant performance degradation with this in my use-case. Another alternative to fixing this issue (I'd imagine) could be by configuring the executorService to not use multiple-threads. However, I'd assume that this is higher impact and more complicated than disabling the CacheLoader. I believe that using the `java.util.concurrent.ConcurrentHashMap` may solve this particular issue. However, I haven't done any further evaluation/analysis of the CacheLoader class to ensure that it is truly safe for use by multiple threads concurrently performaing reads and writes. I would say that the CacheLoader feature seems a bit broken right now in its default use, due to the problem of Drools using multiple threads internally when building the knowledge base. I'm not sure how any of this relates to the Drools v6.x family. -- View this message in context: http://drools.46999.n3.nabble.com/CompositiveClassLoader-CachingLoader-load-method-intermittently-hangs-tp4028780p4029597.html Sent from the Drools: User forum mailing list archive at Nabble.com. _______________________________________________ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users