[rules-users] Problems compiling huge rule base with kie-maven-plugin
Hi all, at work we have a huge rule base (2k rules with an average size of ~2Mib). I've found the building this RB with the latest kie-maven-plugin troublesome at best. I want to share my experience hoping that some Drools developer could help me. I'm using the latest Drools 6.x from the master branch. The first problem I encounterd was that the compilation process retain in memory all the compilation caches until the end. This would lead to a OOM very soon and we can't increase the heap size indefinitely. So the first thing I do was to extract the code from KieModuleMetaInfoBuilder and create a custom builder that would populate a guava cache for every KnowledgeBase compiled. The aforementioned cache would write the bytestreams when the objects get evicted. That worked well and leaded finally to a full compilation. The only problem was the long time one has to wait until completion. Some few rules take up to 30 min. to complete but the majority takes from 5 to 20 secs. So I extended the custom compiler to be multithreaded and finally the compilation time is acceptable again. I also extended the compiler to perform incremental compilations. Now my problem is that the generated kjar is rather huge and when I load it I get an OOM again. Peeking at the code again confirm (if I get this correctly) that all the caches gets loaded altogether, moving the problem to compile time to run time. Is it possible to force the KieContainer to load the caches lazily, based on calls to getKieBase? If not, would it be usefull? I understand that my case is rather on the edge but any help would be appreciated. Thank you very much :) Federico. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Problems compiling huge rule base with kie-maven-plugin
Hi Mario, thanks for the quick reply. Yes I could definitely open a jira and a pull request, just be patient though, as I've to isolate the code from our own. Reproducing the slowdown I experienced would be a little problematic because it depends, I think, entirely on the size (and shape) of our RB. Thanks again, Federico. On 06/18/2014 05:48 PM, Mario Fusco wrote: Hi Federico, so we have 2 problems here. For what regards the compilation one I'd be curious to see your solution. Could you please open a ticket on our jira and attach it there, or maybe send a pull request on github? As for the runtime, yes probably loading stuff lazier in the KieContainer could alleviate also that problem, but first I'd like to give a look at what you're doing to better understand your use case. Could you also attach a reproducer to that jira ticket? Thanks, Mario -- View this message in context: http://drools.46999.n3.nabble.com/Possible-problem-in-PathMemory-tp4030026p4030093.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
