Re: [rules-users] Slow compilation (4h) for a single rule
Thanks for your explanation Edson, it makes sense now I think! Basically what you are saying is that there is a limit, beyond which is no longer feasible to use more ORs on a rule. Edson Tirelli-4 wrote You are trying to avoid the issue. As reported by others, your conditions should be inside the patterns, not in or'd evals in the end, and yes, it is possible to generate rules like that in an automated program. Sadly I don't see how, as some of my rules have arbitrary arithmetic involving more than one variable inside the Number(), for more context see the example I've put on http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td3823232.html. For this case I think I will try to invert the logic and get rid of the ORs: not(A or B) = not(A) and not(B). Best regards, TL -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018894.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
A last effort from my side. Given any number of bound variables a, b, c: to determine whether an arbitrary boolean expression involving arithmetic and comparison operators, a single eval() is sufficient and preferable to a CE combination involving the CE operators and, or, not combining many evals. Instead of not( eval( a == 1 ) ) use eval( ! (a == 1) ) // or better eval( a != 0 ) Instead of eval( a == 1 ) or eval( b == 1 ) use eval( a == 1 || b == 1 ) Instead of eval( a == 1 ) and eval( b == 1 ) use eval( a == 1 b == 1 ) -W On 25/07/2012, fx242 dro...@fx242.com wrote: Thanks for your explanation Edson, it makes sense now I think! Basically what you are saying is that there is a limit, beyond which is no longer feasible to use more ORs on a rule. Edson Tirelli-4 wrote You are trying to avoid the issue. As reported by others, your conditions should be inside the patterns, not in or'd evals in the end, and yes, it is possible to generate rules like that in an automated program. Sadly I don't see how, as some of my rules have arbitrary arithmetic involving more than one variable inside the Number(), for more context see the example I've put on http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td3823232.html. For this case I think I will try to invert the logic and get rid of the ORs: not(A or B) = not(A) and not(B). Best regards, TL -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018894.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
Listen to what others are saying in this thread... it is very good advice, specially what wolfgang suggests bellow regarding collapsing evals. Also, I just read the example in your link where you have multiple accumulates in a single rule... that is *bad* For those that attended or saw my presentation on best practices, an easy way to estimate the performance of accumulates is to remember that the performance of accumulates in a single rule is polinomial: O(n^a), where n is the number of matching facts and a is the number of accumulates you have in the same rule. So, if you have 1 rule with 5 accumulates, the performance will be n^5. If instead you break the accumulates into 5 rules with one accumulate each, you get a performance of O(5n) that is much better. Edson On Wed, Jul 25, 2012 at 7:42 AM, Wolfgang Laun wolfgang.l...@gmail.comwrote: A last effort from my side. Given any number of bound variables a, b, c: to determine whether an arbitrary boolean expression involving arithmetic and comparison operators, a single eval() is sufficient and preferable to a CE combination involving the CE operators and, or, not combining many evals. Instead of not( eval( a == 1 ) ) use eval( ! (a == 1) ) // or better eval( a != 0 ) Instead of eval( a == 1 ) or eval( b == 1 ) use eval( a == 1 || b == 1 ) Instead of eval( a == 1 ) and eval( b == 1 ) use eval( a == 1 b == 1 ) -W On 25/07/2012, fx242 dro...@fx242.com wrote: Thanks for your explanation Edson, it makes sense now I think! Basically what you are saying is that there is a limit, beyond which is no longer feasible to use more ORs on a rule. Edson Tirelli-4 wrote You are trying to avoid the issue. As reported by others, your conditions should be inside the patterns, not in or'd evals in the end, and yes, it is possible to generate rules like that in an automated program. Sadly I don't see how, as some of my rules have arbitrary arithmetic involving more than one variable inside the Number(), for more context see the example I've put on http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td3823232.html . For this case I think I will try to invert the logic and get rid of the ORs: not(A or B) = not(A) and not(B). Best regards, TL -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018894.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli JBoss Drools Core Development JBoss by Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
Thank you all for your feedback, I will try to implement at least the part of collapsing the multiple evals() into a single one. Edson Tirelli-4 wrote Also, I just read the example in your link where you have multiple accumulates in a single rule... that is *bad* The accumulate() over-usage is a known problem for me, but so far the best solution for my case was proposed by Vincent Legendre on this http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-tp3823232p3825280.html post . As a comic note, the previous rule with the 26 ORs, had 26 accumulates on it :) Best regards, TL -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018900.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
[rules-users] Slow compilation (4h) for a single rule
Hi, Recently I was trying to optimize the rule compilation time for my KB (~4K rules). I do a full KB compilation every night, as most of the rules are dynamic generated/converted and managed by the business operators. The compile time was not very good (around 3 hours on a 8 core Xeon CPU), but is was acceptable. On the other night I was alarmed that the compile time was doubled overnight without an obvious explanation or any relevant rule count increase. After some debuging I've traced the problem and was shocked to find that most of the rules KB (4K rules) compile under 5 minutes, except one. This single rule was taking all the 4h to compile! I'm using DROOLS 5.2 Final. The rule example below samples the problem (don't mind the nonsense Number() conditions): rule CONFIG_1-3UCO07 salience -90 when Number(rel_4226982244: intValue) Number(rel_7521194: intValue) Number(rel_787633980: intValue) Number(qty_1331544548: intValue) Number(rel_1425187049: intValue) Number(rel_1180441096: intValue) Number(rel_3132221704: intValue) Number(rel_1663554156: intValue) Number(rel_1940612775: intValue) Number(rel_1735126416: intValue) Number(rel_3962361266: intValue) Number(rel_882187: intValue) Number(rel_1169008280: intValue) Number(rel_3495503197: intValue) Number(rel_70290066: intValue) Number(rel_1333860961: intValue) Number(rel_2793542368: intValue) Number(rel_952404632: intValue) Number(rel_2712335119: intValue) Number(qty_4276673135: intValue) Number(qty_3950051097: intValue) Number(rel_3391032645: intValue) Number(rel_2738029181: intValue) Number(qty_4125201080: intValue) Number(rel_663254919: intValue) Number(rel_3059142355: intValue) not((eval(qty_1331544548 == 1) or eval(rel_1425187049 == 1) or eval(rel_1180441096 == 1) or eval(rel_787633980 = 1) or eval(rel_7521194 = 1) or eval(rel_4226982244 = 1) or eval(rel_2712335119 == 1) or eval(qty_3950051097 == 1) or eval(qty_4276673135 == 1) or eval(rel_1940612775 == 1) or eval(rel_3132221704 == 1) or eval(rel_882187 == 1) or eval(rel_1663554156 == 1) or eval(rel_1735126416 == 1) or eval(rel_3962361266 == 1) or eval(rel_1169008280 == 1) or eval(rel_2793542368 == 1) or eval(rel_3495503197 == 1) or eval(qty_4125201080 == 1) or eval(rel_70290066 == 1) or eval(rel_3391032645 == 1) or eval(rel_1333860961 == 1) or eval(rel_2738029181 == 1) or eval(rel_663254919 == 1) or eval(rel_952404632 == 1) or eval(rel_3059142355 == 1))) then vh.error(kcontext, Error ...); end I've noticed that the last not(...) is the problematic condition. Is there a limit using eval() and or-clauses, or a known problem? Regards, Tiago Lopes -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
On 24/07/2012, fx242 dro...@fx242.com wrote: Hi, This single rule was taking all the 4h to compile! I'm using DROOLS 5.2 Final. The rule example below samples the problem (don't mind the nonsense Number() conditions): Well, I do mind the nonsense Number(). There's no point in discussing this if you don't post an exact image of your rule. -W ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
laune wrote Well, I do mind the nonsense Number(). There's no point in discussing this if you don't post an exact image of your rule. -W There is a point, because the rule I've posted compiles and causes the same problem, I've just removed redundant code to expose the problematic section. TL -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018857.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
As far as I remember, there were some fixes around compilation time for edge cases in 5.4. Could you please give a try to 5.4 or even 5.5-SNAPSHOT? Best Regards, Esteban Aliverti - Blog @ http://ilesteban.wordpress.com On Tue, Jul 24, 2012 at 1:11 PM, Wolfgang Laun wolfgang.l...@gmail.comwrote: On 24/07/2012, fx242 dro...@fx242.com wrote: Hi, This single rule was taking all the 4h to compile! I'm using DROOLS 5.2 Final. The rule example below samples the problem (don't mind the nonsense Number() conditions): Well, I do mind the nonsense Number(). There's no point in discussing this if you don't post an exact image of your rule. -W ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
There are 26 patterns Number(). Now I don't know how many facts of that type you have, but this sequence will produce all combinations of all available facts. (This is the worst example of this anti-pattern I've seen.) The or-ed evals will try to create parallel nodes in the network, and the not should guarantee that none of those exists. I'm not sure what the rule should actually ascertain. Could you provide a simple example, using 3 numbers, describing verbally or by a regular boolean expression when the rule should fire? E.g. it should fire if there is one Number != 1, another Number 1 and a third Number != 1 -W On 24/07/2012, Esteban Aliverti esteban.alive...@gmail.com wrote: As far as I remember, there were some fixes around compilation time for edge cases in 5.4. Could you please give a try to 5.4 or even 5.5-SNAPSHOT? Best Regards, Esteban Aliverti - Blog @ http://ilesteban.wordpress.com On Tue, Jul 24, 2012 at 1:11 PM, Wolfgang Laun wolfgang.l...@gmail.comwrote: On 24/07/2012, fx242 dro...@fx242.com wrote: Hi, This single rule was taking all the 4h to compile! I'm using DROOLS 5.2 Final. The rule example below samples the problem (don't mind the nonsense Number() conditions): Well, I do mind the nonsense Number(). There's no point in discussing this if you don't post an exact image of your rule. -W ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
Hi, There is something that I must ask... Why do you evaluate perfectly simple patterns in an eval, I wonder? rule CONFIG_1-3UCO07 salience -90 when Number(rel_4226982244: intValue 1 ) Number(rel_7521194: intValue 1 ) Number(rel_787633980: intValue 1 ) Number(qty_1331544548: intValue != 1 ) Number(rel_1425187049: intValue != 1 ) Number(rel_1180441096: intValue != 1 ) Number(rel_3132221704: intValue != 1 ) Number(rel_1663554156: intValue != 1 ) Number(rel_1940612775: intValue != 1 ) Number(rel_1735126416: intValue != 1 ) Number(rel_3962361266: intValue != 1 ) Number(rel_882187: intValue != 1 ) Number(rel_1169008280: intValue != 1 ) Number(rel_3495503197: intValue != 1 ) Number(rel_70290066: intValue != 1 ) Number(rel_1333860961: intValue != 1 ) Number(rel_2793542368: intValue != 1 ) Number(rel_952404632: intValue != 1 ) Number(rel_2712335119: intValue != 1 ) Number(qty_4276673135: intValue != 1 ) Number(qty_3950051097: intValue != 1 ) Number(rel_3391032645: intValue != 1 ) Number(rel_2738029181: intValue != 1 ) Number(qty_4125201080: intValue != 1 ) Number(rel_663254919: intValue != 1 ) Number(rel_3059142355: intValue != 1 ) then vh.error(kcontext, Error ...); end Of course, this is still nonsense as long as you do not have a way to constrain the number that you evaluate. The rule, as written above, will fire from the moment you have one single number that has an intValue below 1. Regards, Frank -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018860.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
I understand that the rule looks pointless and under-optimized, but it's the only way I could represent this kind of rule in an automated way (I didn't write the rule by hand). Basically with these rules I try to pre-compute the amount of some products on the WM (using count() inside a accumulate function) resulting in a Number(), and then on the last condition I evaluate some arbitrary arithmetic operation using the variables inside the Number() object. For more details for examples see my previous post where I've tried to address this issue: http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td3823232.html So, considerations apart of the way the rule is written, I still think there is a bug or something similar on the RETE compiler, that causes this rule to be processed in a very expensive way. Can someone confirm this, and try to sugest an alternative? Maybe It's something that is already fixed on newer versions as sugested by Esteban? Regards, Tiago Lopes -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018870.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Slow compilation (4h) for a single rule
You are trying to avoid the issue. As reported by others, your conditions should be inside the patterns, not in or'd evals in the end, and yes, it is possible to generate rules like that in an automated program. Now, the reason it is taking so long is because of the ors. I don't think there is any bug there. Just for you to understand, an OR is just syntax sugar for multiple rules. So if you have a rule that says: A() or B() The engine is actually creating 2 rules: rule 1 when A() ... rule 2 when B() ... I.e., it generates all possible combinations of rules, one for each of your logical branches. Now, I quickly counted 26 ors in your rule. It is quite possible that the engine is generating all possible combinations of rules for all your ORs, and that is roughly (there are some factors that influence the number of rules): 2^26 == 67108864 rules. To compile 67 million rules, it can indeed take some time. I don't know for sure if that is the case without looking at your application, but to solve your problem you should fix your rule generation, not look for a problem elsewhere. Edson On Tue, Jul 24, 2012 at 9:29 AM, fx242 dro...@fx242.com wrote: I understand that the rule looks pointless and under-optimized, but it's the only way I could represent this kind of rule in an automated way (I didn't write the rule by hand). Basically with these rules I try to pre-compute the amount of some products on the WM (using count() inside a accumulate function) resulting in a Number(), and then on the last condition I evaluate some arbitrary arithmetic operation using the variables inside the Number() object. For more details for examples see my previous post where I've tried to address this issue: http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td3823232.html So, considerations apart of the way the rule is written, I still think there is a bug or something similar on the RETE compiler, that causes this rule to be processed in a very expensive way. Can someone confirm this, and try to sugest an alternative? Maybe It's something that is already fixed on newer versions as sugested by Esteban? Regards, Tiago Lopes -- View this message in context: http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-tp4018855p4018870.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli JBoss Drools Core Development JBoss by Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
