[rules-users] Shadow facts(was JRules\Drools benchmarking)
I did some reading about shadow facts and I was thinking about turning them off, but it seems to be that if I use stateless sessions (session.execute()) that shadow facts are irrelevant. Is this true? _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Edson Tirelli Sent: Thursday, May 15, 2008 10:00 AM To: Rules Users List Subject: Re: [rules-users] JRules\Drools benchmarking... It seems you are using a good strategy to do your tests. But still, it is difficult to explain why one is slower than the other without seeing the actual test code. This is because all the engines have stronger and weaker spots. Just to mention one example, some engines (not talking specifically about drools and jrules, but about all engines) implement faster alpha evaluation, others implement faster beta (join) evaluation, others implement good optimizations for not() while others may focus on eval(), etc. It is up to the point that when comparing 2 engines, one performs better in hardware with a bigger L2 cache while the other performs better in hardware with a smaller L2 cache. So, best I can do without looking at the actual tests is provide you some tips: 1. First of all, are you using Drools 4.0.7? It is very important that you use this version over the previous ones. 2. Are you using stateful or stateless sessions? If you are using stateful sessions are you calling dispose() after using the session? If not, you are inflating your memory and certainly causing the engine to run slower over time. 3. Are you sharing the rulebase among multiple requests? The drools rulebase is designed to be shared and the compilation process is eager and pretty heavy compared to session creation. So, it pays off to create the rulebase once and share among requests. 4. Did you disabled shadow facts? Test cases usually use a really small fact base, so would not be much affected by shadow facts, but still, disabling them improves performance, but require some best practices to be followed. 5. Do your rules follow best practices (similar to SQL writing best practices), i.e., write the most constraining patterns first, write the most constraining restrictions first, etc? Do you write patterns in the same order among rules to maximize node sharing? I guess you do, but worth mentioning anyway. Anyway, just some tips. Regarding the jrules blog, I know it, but I make a bet with you. Download the manners benchmark to your machine, make sure the rules are the correct ones (not cheated ones), run the test on both engines and share the results. I pay you a beer if you get results similar to those published in the blog. :) My point is not that we are faster (what I know we are) or them are faster. My point is that perf benchmarks for rules engines are a really tricky matter, with lots of variables involved, that make every test case configuration unique. Try to reproduce in a different environment and you will get different performance rates between the engines. That is why, our recommendation is to always do what you are doing: try your own use case. Now, whatever you are trying, I'm sure it is possible to optimize if we see the test case, but is it worth it? Or the perf as it is already meets your requirements? Cheers, Edson PS: I'm serious about the beer... ;) run and share the results with us... 2008/5/15 mmquelo massi [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] : You r right... I have to tell you what I have done... I did not define a stand-alone benchmark like the Manners one. I benchmarked a real j2ee application. I have got jrules deployed with a resource adapter and drools deployed with simple jars libraries plus jbrms. Jrules uses a bres module which does the same trick jbrms does. Both of them are deployed on the same AS, in the same time, same machine (my laptop: dual core 2 duo 1.66, 2GB). Using the inversion of control pattern I found out how to switch the rule engine at run-time. So I can easily choose then rule engine to use between drools and jrules. Ofcourse thay have got two separate rule repositories but both of them persist the rules on the same DB which is Derby. The j2ee application I benchmarked sends a request object to the current rule engine and get back a reply from it. I just measured the elapsed time between the request and reply generation using drools first and the jrules. I did the measurements tens of times. Both rule engines implement the same rules and the Drools rules (which I personally implemented) are at least as optimized as the jrules ones. In the Jrules version of the rules there are a lot of Eval(...) blocks in the Drools version I did not use the Eval() at all but I just did pattern matching. If you want i can send you a more specific documentation but I hope this explanation will be enough to show you that the measurements I have done are not that bad. In any case I noticed that after a warming-up phase, the drools engine
Re: [rules-users] Shadow facts(was JRules\Drools benchmarking)
Hehl, Thomas wrote: I did some reading about shadow facts and I was thinking about turning them off, but it seems to be that if I use stateless sessions (session.execute()) that shadow facts are irrelevant. Is this true? No, only if you turn on sequential mode. Mark *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *Edson Tirelli *Sent:* Thursday, May 15, 2008 10:00 AM *To:* Rules Users List *Subject:* Re: [rules-users] JRules\Drools benchmarking... It seems you are using a good strategy to do your tests. But still, it is difficult to explain why one is slower than the other without seeing the actual test code. This is because all the engines have stronger and weaker spots. Just to mention one example, some engines (not talking specifically about drools and jrules, but about all engines) implement faster alpha evaluation, others implement faster beta (join) evaluation, others implement good optimizations for not() while others may focus on eval(), etc. It is up to the point that when comparing 2 engines, one performs better in hardware with a bigger L2 cache while the other performs better in hardware with a smaller L2 cache. So, best I can do without looking at the actual tests is provide you some tips: 1. First of all, are you using Drools 4.0.7? It is very important that you use this version over the previous ones. 2. Are you using stateful or stateless sessions? If you are using stateful sessions are you calling dispose() after using the session? If not, you are inflating your memory and certainly causing the engine to run slower over time. 3. Are you sharing the rulebase among multiple requests? The drools rulebase is designed to be shared and the compilation process is eager and pretty heavy compared to session creation. So, it pays off to create the rulebase once and share among requests. 4. Did you disabled shadow facts? Test cases usually use a really small fact base, so would not be much affected by shadow facts, but still, disabling them improves performance, but require some best practices to be followed. 5. Do your rules follow best practices (similar to SQL writing best practices), i.e., write the most constraining patterns first, write the most constraining restrictions first, etc? Do you write patterns in the same order among rules to maximize node sharing? I guess you do, but worth mentioning anyway. Anyway, just some tips. Regarding the jrules blog, I know it, but I make a bet with you. Download the manners benchmark to your machine, make sure the rules are the correct ones (not cheated ones), run the test on both engines and share the results. I pay you a beer if you get results similar to those published in the blog. :) My point is not that we are faster (what I know we are) or them are faster. My point is that perf benchmarks for rules engines are a really tricky matter, with lots of variables involved, that make every test case configuration unique. Try to reproduce in a different environment and you will get different performance rates between the engines. That is why, our recommendation is to always do what you are doing: try your own use case. Now, whatever you are trying, I'm sure it is possible to optimize if we see the test case, but is it worth it? Or the perf as it is already meets your requirements? Cheers, Edson PS: I'm serious about the beer... ;) run and share the results with us... 2008/5/15 mmquelo massi [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: You r right... I have to tell you what I have done... I did not define a stand-alone benchmark like the Manners one. I benchmarked a real j2ee application. I have got jrules deployed with a resource adapter and drools deployed with simple jars libraries plus jbrms. Jrules uses a bres module which does the same trick jbrms does. Both of them are deployed on the same AS, in the same time, same machine (my laptop: dual core 2 duo 1.66, 2GB). Using the inversion of control pattern I found out how to switch the rule engine at run-time. So I can easily choose then rule engine to use between drools and jrules. Ofcourse thay have got two separate rule repositories but both of them persist the rules on the same DB which is Derby. The j2ee application I benchmarked sends a request object to the current rule engine and get back a reply from it. I just measured the elapsed time between the request and reply generation using drools first and the jrules. I did the measurements tens of times. Both rule engines implement the same rules and the Drools rules (which I personally implemented) are at least as optimized as the jrules ones. In the Jrules version of the rules there are a lot of Eval(...) blocks in the Drools version I did not use the Eval() at all but I just did pattern matching. If you want i can send you a more specific
RE: [rules-users] Shadow facts(was JRules\Drools benchmarking)
OK, I did. So that means I need to disable shadow facts? _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mark Proctor Sent: Thursday, May 15, 2008 12:22 PM To: Rules Users List Subject: Re: [rules-users] Shadow facts(was JRules\Drools benchmarking) Hehl, Thomas wrote: I did some reading about shadow facts and I was thinking about turning them off, but it seems to be that if I use stateless sessions (session.execute()) that shadow facts are irrelevant. Is this true? No, only if you turn on sequential mode. Mark _ From: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ] On Behalf Of Edson Tirelli Sent: Thursday, May 15, 2008 10:00 AM To: Rules Users List Subject: Re: [rules-users] JRules\Drools benchmarking... It seems you are using a good strategy to do your tests. But still, it is difficult to explain why one is slower than the other without seeing the actual test code. This is because all the engines have stronger and weaker spots. Just to mention one example, some engines (not talking specifically about drools and jrules, but about all engines) implement faster alpha evaluation, others implement faster beta (join) evaluation, others implement good optimizations for not() while others may focus on eval(), etc. It is up to the point that when comparing 2 engines, one performs better in hardware with a bigger L2 cache while the other performs better in hardware with a smaller L2 cache. So, best I can do without looking at the actual tests is provide you some tips: 1. First of all, are you using Drools 4.0.7? It is very important that you use this version over the previous ones. 2. Are you using stateful or stateless sessions? If you are using stateful sessions are you calling dispose() after using the session? If not, you are inflating your memory and certainly causing the engine to run slower over time. 3. Are you sharing the rulebase among multiple requests? The drools rulebase is designed to be shared and the compilation process is eager and pretty heavy compared to session creation. So, it pays off to create the rulebase once and share among requests. 4. Did you disabled shadow facts? Test cases usually use a really small fact base, so would not be much affected by shadow facts, but still, disabling them improves performance, but require some best practices to be followed. 5. Do your rules follow best practices (similar to SQL writing best practices), i.e., write the most constraining patterns first, write the most constraining restrictions first, etc? Do you write patterns in the same order among rules to maximize node sharing? I guess you do, but worth mentioning anyway. Anyway, just some tips. Regarding the jrules blog, I know it, but I make a bet with you. Download the manners benchmark to your machine, make sure the rules are the correct ones (not cheated ones), run the test on both engines and share the results. I pay you a beer if you get results similar to those published in the blog. :) My point is not that we are faster (what I know we are) or them are faster. My point is that perf benchmarks for rules engines are a really tricky matter, with lots of variables involved, that make every test case configuration unique. Try to reproduce in a different environment and you will get different performance rates between the engines. That is why, our recommendation is to always do what you are doing: try your own use case. Now, whatever you are trying, I'm sure it is possible to optimize if we see the test case, but is it worth it? Or the perf as it is already meets your requirements? Cheers, Edson PS: I'm serious about the beer... ;) run and share the results with us... 2008/5/15 mmquelo massi [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] : You r right... I have to tell you what I have done... I did not define a stand-alone benchmark like the Manners one. I benchmarked a real j2ee application. I have got jrules deployed with a resource adapter and drools deployed with simple jars libraries plus jbrms. Jrules uses a bres module which does the same trick jbrms does. Both of them are deployed on the same AS, in the same time, same machine (my laptop: dual core 2 duo 1.66, 2GB). Using the inversion of control pattern I found out how to switch the rule engine at run-time. So I can easily choose then rule engine to use between drools and jrules. Ofcourse thay have got two separate rule repositories but both of them persist the rules on the same DB which is Derby. The j2ee application I benchmarked sends a request object to the current rule engine and get back a reply from it. I just measured the elapsed time between the request and reply generation using drools first and the jrules. I did the measurements tens of times. Both rule engines implement the same rules and the Drools rules (which I personally implemented) are at least as optimized
Re: [rules-users] Shadow facts(was JRules\Drools benchmarking)
Hehl, Thomas wrote: OK, I did. So that means I need to disable shadow facts? sequential mode already does, as there is no inference modify doesn't do anything and thus shadow facts aren't needed. Mark *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *Mark Proctor *Sent:* Thursday, May 15, 2008 12:22 PM *To:* Rules Users List *Subject:* Re: [rules-users] Shadow facts(was JRules\Drools benchmarking) Hehl, Thomas wrote: I did some reading about shadow facts and I was thinking about turning them off, but it seems to be that if I use stateless sessions (session.execute()) that shadow facts are irrelevant. Is this true? No, only if you turn on sequential mode. Mark *From:* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *Edson Tirelli *Sent:* Thursday, May 15, 2008 10:00 AM *To:* Rules Users List *Subject:* Re: [rules-users] JRules\Drools benchmarking... It seems you are using a good strategy to do your tests. But still, it is difficult to explain why one is slower than the other without seeing the actual test code. This is because all the engines have stronger and weaker spots. Just to mention one example, some engines (not talking specifically about drools and jrules, but about all engines) implement faster alpha evaluation, others implement faster beta (join) evaluation, others implement good optimizations for not() while others may focus on eval(), etc. It is up to the point that when comparing 2 engines, one performs better in hardware with a bigger L2 cache while the other performs better in hardware with a smaller L2 cache. So, best I can do without looking at the actual tests is provide you some tips: 1. First of all, are you using Drools 4.0.7? It is very important that you use this version over the previous ones. 2. Are you using stateful or stateless sessions? If you are using stateful sessions are you calling dispose() after using the session? If not, you are inflating your memory and certainly causing the engine to run slower over time. 3. Are you sharing the rulebase among multiple requests? The drools rulebase is designed to be shared and the compilation process is eager and pretty heavy compared to session creation. So, it pays off to create the rulebase once and share among requests. 4. Did you disabled shadow facts? Test cases usually use a really small fact base, so would not be much affected by shadow facts, but still, disabling them improves performance, but require some best practices to be followed. 5. Do your rules follow best practices (similar to SQL writing best practices), i.e., write the most constraining patterns first, write the most constraining restrictions first, etc? Do you write patterns in the same order among rules to maximize node sharing? I guess you do, but worth mentioning anyway. Anyway, just some tips. Regarding the jrules blog, I know it, but I make a bet with you. Download the manners benchmark to your machine, make sure the rules are the correct ones (not cheated ones), run the test on both engines and share the results. I pay you a beer if you get results similar to those published in the blog. :) My point is not that we are faster (what I know we are) or them are faster. My point is that perf benchmarks for rules engines are a really tricky matter, with lots of variables involved, that make every test case configuration unique. Try to reproduce in a different environment and you will get different performance rates between the engines. That is why, our recommendation is to always do what you are doing: try your own use case. Now, whatever you are trying, I'm sure it is possible to optimize if we see the test case, but is it worth it? Or the perf as it is already meets your requirements? Cheers, Edson PS: I'm serious about the beer... ;) run and share the results with us... 2008/5/15 mmquelo massi [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: You r right... I have to tell you what I have done... I did not define a stand-alone benchmark like the Manners one. I benchmarked a real j2ee application. I have got jrules deployed with a resource adapter and drools deployed with simple jars libraries plus jbrms. Jrules uses a bres module which does the same trick jbrms does. Both of them are deployed on the same AS, in the same time, same machine (my laptop: dual core 2 duo 1.66, 2GB). Using the inversion of control pattern I found out how to switch the rule engine at run-time. So I can easily choose then rule engine to use between drools and jrules. Ofcourse thay have got two separate rule repositories but both of them persist the rules on the same DB which is Derby. The j2ee application I benchmarked sends a request object to the current
