Re: [rules-users] Rule compilation errors under heavy load
I've filed a JIRA for this issue http://jira.jboss.com/jira/browse/JBRULES-1114 ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
On 7/14/07, Mark Proctor [EMAIL PROTECTED] wrote:
We are going to need an integration test, to reproduce this, can you
supply one?
Okay, I've just converted the simple test that I added earlier to a
Junit test. The test is not guaranteed to fail if there are thread
safety errors, just very likely to fail (don't know if this is
acceptable for you). Initially, I wrote a test which guaranteed that
calls to the addPackage() and getPackage() methods were interleaved by
different threads, but, annoyingly, this test passed. It seems that a
context-switch halfway through one of these methods is necessary to
provoke the problem (which is difficult to guarantee :-)
Dean.
package org.drools.integrationtests;
import java.util.ArrayList;
import java.util.List;
import junit.framework.TestCase;
import org.drools.compiler.PackageBuilderConfiguration;
import org.drools.compiler.PackageBuilderErrors;
import org.drools.lang.descr.FunctionDescr;
import org.drools.lang.descr.ImportDescr;
import org.drools.lang.descr.PackageDescr;
public class PackageBuilderThreadSafetyTest extends TestCase {
private static final int _NUMBER_OF_THREADS = 100;
private static final int _SLEEP_TIME_MS = 100;
public void testThreadSafety() {
final ListPackageBuilderErrors errors = new ArrayListPackageBuilderErrors();
final ListException exceptions= new ArrayListException();
Thread[] threads = new Thread[_NUMBER_OF_THREADS];
for (int i = 0; i _NUMBER_OF_THREADS; i++) {
Thread testThread = new Thread() {
public void run() {
try {
PackageBuilderConfiguration packageBuilderConfig = new PackageBuilderConfiguration();
org.drools.compiler.PackageBuilder builder = null;
try {
builder = new org.drools.compiler.PackageBuilder(packageBuilderConfig);
} catch (Throwable t) {
t.printStackTrace();
throw new RuntimeException(t);
}
PackageDescr packageDescr = new PackageDescr(MyRulebase);
addImports(packageDescr);
addFunctions(packageDescr);
// added some arbitrary sleep statements to encourage
// context switching and hope this provokes exceptions
sleep(_SLEEP_TIME_MS);
builder.addPackage(packageDescr);
sleep(_SLEEP_TIME_MS);
builder.getPackage();
sleep(_SLEEP_TIME_MS);
if (builder.hasErrors()) {
errors.add(builder.getErrors());
}
} catch (Exception e) {
e.printStackTrace();
exceptions.add(e);
}
}
};
threads[i] = testThread;
try {
testThread.start();
} catch (Exception e) {
assertTrue(false);
}
}
for (int i = 0; i _NUMBER_OF_THREADS; i++) {
try {
threads[i].join();
} catch (InterruptedException e) {
threads[i].interrupt();
}
}
assertTrue(Exceptions during package compilation (number= + exceptions.size() + ) , exceptions.isEmpty());
assertTrue(PackageBuilderErrors during package compilation (number= + errors.size() + ), errors.isEmpty());
}
private static void addImports(PackageDescr packageDescr) {
packageDescr.addImport(new ImportDescr(java.util.List));
packageDescr.addImport(new ImportDescr(java.util.ArrayList));
packageDescr.addImport(new ImportDescr(java.util.LinkedList));
packageDescr.addImport(new ImportDescr(java.util.Set));
packageDescr.addImport(new ImportDescr(java.util.HashSet));
packageDescr.addImport(new ImportDescr(java.util.SortedSet));
packageDescr.addImport(new ImportDescr(java.util.TreeSet));
}
private static void addFunctions(PackageDescr packageDescr) {
FunctionDescr functionDescr = new FunctionDescr(foo, void);
functionDescr.addParameter(String, arg1);
String body =
Set myHashSet = new HashSet(); +
myHashSet.add(arg1); +
List myArrayList = new ArrayList(); +
myArrayList.add(arg1); +
List myLinkedList = new LinkedList(); +
myLinkedList.add(arg1); +
Set myTreeSet = new TreeSet(); +
myTreeSet.add(arg1);;
functionDescr.setText(body);
packageDescr.addFunction(functionDescr);
}
}
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
RE: [rules-users] Rule compilation errors under heavy load
Couldn't you use yield() to generate a context-switch? It would only happen in that spot, but should produce some good testing. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dean Jones Sent: Monday, July 16, 2007 5:48 AM To: Rules Users List Subject: Re: [rules-users] Rule compilation errors under heavy load On 7/14/07, Mark Proctor [EMAIL PROTECTED] wrote: We are going to need an integration test, to reproduce this, can you supply one? Okay, I've just converted the simple test that I added earlier to a Junit test. The test is not guaranteed to fail if there are thread safety errors, just very likely to fail (don't know if this is acceptable for you). Initially, I wrote a test which guaranteed that calls to the addPackage() and getPackage() methods were interleaved by different threads, but, annoyingly, this test passed. It seems that a context-switch halfway through one of these methods is necessary to provoke the problem (which is difficult to guarantee :-) Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
Hi Thomas, On 7/16/07, Hehl, Thomas [EMAIL PROTECTED] wrote: Couldn't you use yield() to generate a context-switch? It would only happen in that spot, but should produce some good testing. It's a good point about using yield(). I did use this originally, but for some reason it generated fewer errors than using some arbitrary sleep values. At the time, I explained this to myself as follows: yield() means that context switching happens at well-defined points, whereas when there are lots of sleeping threads waking up at arbitrary moments wanting to be run, context switching happens more randomly. I'm not sure if this explanation makes complete sense, though. Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
RE: [rules-users] Rule compilation errors under heavy load
Yeah, me either. Welcome to Java!:) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dean Jones Sent: Monday, July 16, 2007 9:09 AM To: Rules Users List Subject: Re: [rules-users] Rule compilation errors under heavy load Hi Thomas, On 7/16/07, Hehl, Thomas [EMAIL PROTECTED] wrote: Couldn't you use yield() to generate a context-switch? It would only happen in that spot, but should produce some good testing. It's a good point about using yield(). I did use this originally, but for some reason it generated fewer errors than using some arbitrary sleep values. At the time, I explained this to myself as follows: yield() means that context switching happens at well-defined points, whereas when there are lots of sleeping threads waking up at arbitrary moments wanting to be run, context switching happens more randomly. I'm not sure if this explanation makes complete sense, though. Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
We are going to need an integration test, to reproduce this, can you supply one? Mark Dean Jones wrote: Hi Mark, On 7/13/07, Mark Proctor [EMAIL PROTECTED] wrote: I think I've alread fixed this in trunk - when dealing with add/removing stuff I now do a block syncronisation on the pkgs instance. If that test is clean, we should add it to the list of integration tests, to track regressions. I just tried running my test against a build from the trunk (after faffing around with maven for a while) and I still get varying numbers of mvel exceptions and function compilation errors. Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
I think I've alread fixed this in trunk - when dealing with add/removing
stuff I now do a block syncronisation on the pkgs instance. If that test
is clean, we should add it to the list of integration tests, to track
regressions.
Mark
Edson Tirelli wrote:
Dean
Thanks, we will investigate that asap.
[]s
Edson
2007/7/12, Dean Jones [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:
Hi Edson,
On 7/12/07, Edson Tirelli [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] wrote:
Is it possible for you to provide a test case capable of
reproducing the
problem?
I've written a quick test case, which is attached. When
numberOfThreads==1,
everything compiles okay. When numberOfThreads==100, I get function
compilation errors and CompileExceptions from mvel. Clearly there is
nothing shared between these threads as a new test class is created
per thread. If you synchronise the calls to addPackage() and
getPackage() using the class monitor i.e.
synchronized (DroolsLoadTest.class) {
builder.addPackage(packageDescr);
builder.getPackage();
}
then the problems go away. Hope this helps.
At the same time, I would suggest you to not follow such
approach in your
application. Compilation is an extremely heavy process and
should not be
done for each request. The ideal approach is to pre-compile (for
instance,
we have a contributed ant-task for that) and serialize your
rulebase to your
database or filesystem, or at least, compile it only once and
cache it at
runtime.
That's very sensible advice. Our situation is slightly different as
users are allowed to change the rules used by a running service,
so we
need to reflect any changes when a new request comes in. We are
planning to apply caching if performance is an issue, but we were
trying to assess performance of application under load when this
thread safety issue arose.
Thanks for your help,
Dean.
___
rules-users mailing list
rules-users@lists.jboss.org mailto:rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
Software Engineer - JBoss Rules Core Developer
Office: +55 11 3529-6000
Mobile: +55 11 9287-5646
JBoss, a division of Red Hat @ www.jboss.com http://www.jboss.com
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
Hi Mark, On 7/13/07, Mark Proctor [EMAIL PROTECTED] wrote: I think I've alread fixed this in trunk - when dealing with add/removing stuff I now do a block syncronisation on the pkgs instance. If that test is clean, we should add it to the list of integration tests, to track regressions. I just tried running my test against a build from the trunk (after faffing around with maven for a while) and I still get varying numbers of mvel exceptions and function compilation errors. Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
Is it possible for you to provide a test case capable of reproducing the problem? At the same time, I would suggest you to not follow such approach in your application. Compilation is an extremely heavy process and should not be done for each request. The ideal approach is to pre-compile (for instance, we have a contributed ant-task for that) and serialize your rulebase to your database or filesystem, or at least, compile it only once and cache it at runtime. []s Edson 2007/7/12, Dean Jones [EMAIL PROTECTED]: Hello folks, I'm experiencing some odd behaviour from Drools (or maybe the Eclipse compiler) when load-testing my application, and wondered if anyone else had experienced the same issue. I have a web service which, for every request, loads in rules from the database, creates a org.drools.lang.descr.PackageDescr and then uses this to create a org.drools.rule.Package. The code is something like: PackageBuilderConfiguration conf = new PackageBuilderConfiguration(); PackageBuilder builder = new PackageBuilder(conf); PackageDescr packageDescr = new PackageDescr(MyRulebaseName); populatePackageDescr(packageDescr); builder.addPackage(packageDescr); Package _package = builder.getPackage(); return _package; This code gets run every request. The populatePackageDescr() method adds imports, globals, functions and rules to the PackageDescr. My load-test just repeatedly calls the same rulebase, so populatePackageDescr() will be doing exactly the same thing for each request. Occasionally (normally once or twice per 100 requests) I get a rule compilation error (the exact error varies from request to request) whereas most of the time the same rule compiles fine. This makes me suspect a thread-safety issue in either the PackageBuilder or the compiler (I'm using the eclipse compiler). Initially, I suspected a problem in my populatePackageDescr(packageDescr) method, but I think I've ruled this out. There is nothing stateful in this method, and I've tried dumping the XML representation of the packageDescr immediately after populating it; it looks fine to me and is exactly the same whether compilation succeeds or and fails. Has anyone done this kind of thing successfully under a heavy load? Thanks, Dean. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- Edson Tirelli Software Engineer - JBoss Rules Core Developer Office: +55 11 3529-6000 Mobile: +55 11 9287-5646 JBoss, a division of Red Hat @ www.jboss.com ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Rule compilation errors under heavy load
Hi Edson,
On 7/12/07, Edson Tirelli [EMAIL PROTECTED] wrote:
Is it possible for you to provide a test case capable of reproducing the
problem?
I've written a quick test case, which is attached. When numberOfThreads==1,
everything compiles okay. When numberOfThreads==100, I get function
compilation errors and CompileExceptions from mvel. Clearly there is
nothing shared between these threads as a new test class is created
per thread. If you synchronise the calls to addPackage() and
getPackage() using the class monitor i.e.
synchronized (DroolsLoadTest.class) {
builder.addPackage(packageDescr);
builder.getPackage();
}
then the problems go away. Hope this helps.
At the same time, I would suggest you to not follow such approach in your
application. Compilation is an extremely heavy process and should not be
done for each request. The ideal approach is to pre-compile (for instance,
we have a contributed ant-task for that) and serialize your rulebase to your
database or filesystem, or at least, compile it only once and cache it at
runtime.
That's very sensible advice. Our situation is slightly different as
users are allowed to change the rules used by a running service, so we
need to reflect any changes when a new request comes in. We are
planning to apply caching if performance is an issue, but we were
trying to assess performance of application under load when this
thread safety issue arose.
Thanks for your help,
Dean.
import org.drools.compiler.DroolsError;
import org.drools.compiler.PackageBuilderConfiguration;
import org.drools.compiler.PackageBuilderErrors;
import org.drools.lang.descr.FunctionDescr;
import org.drools.lang.descr.ImportDescr;
import org.drools.lang.descr.PackageDescr;
public class DroolsLoadTest {
public DroolsLoadTest() {
}
public void runTest() {
PackageBuilderConfiguration packageBuilderConfig = new PackageBuilderConfiguration();
org.drools.compiler.PackageBuilder builder = new org.drools.compiler.PackageBuilder(packageBuilderConfig);
PackageDescr packageDescr = new PackageDescr(MyRulebase);
addImports(packageDescr);
addFunctions(packageDescr);
builder.addPackage(packageDescr);
builder.getPackage();
if (builder.hasErrors()) {
PackageBuilderErrors errors = builder.getErrors();
for (DroolsError error : errors.getErrors()) {
System.out.println(ERROR: + error.getMessage());
}
}
}
private void addImports(PackageDescr packageDescr) {
packageDescr.addImport(new ImportDescr(java.util.List));
packageDescr.addImport(new ImportDescr(java.util.ArrayList));
packageDescr.addImport(new ImportDescr(java.util.LinkedList));
packageDescr.addImport(new ImportDescr(java.util.Set));
packageDescr.addImport(new ImportDescr(java.util.HashSet));
packageDescr.addImport(new ImportDescr(java.util.SortedSet));
packageDescr.addImport(new ImportDescr(java.util.TreeSet));
}
private void addFunctions(PackageDescr packageDescr) {
FunctionDescr functionDescr = new FunctionDescr(foo, void);
functionDescr.addParameter(String, arg1);
String body =
Set myHashSet = new HashSet(); +
myHashSet.add(arg1); +
List myArrayList = new ArrayList(); +
myArrayList.add(arg1); +
List myLinkedList = new LinkedList(); +
myLinkedList.add(arg1); +
Set myTreeSet = new TreeSet(); +
myTreeSet.add(arg1);;
functionDescr.setText(body);
packageDescr.addFunction(functionDescr);
}
public static void main(String[] args) {
int numberOfThreads = 100;
Thread[] threads = new Thread[numberOfThreads];
for (int i = 0; i numberOfThreads; i++) {
Thread testThread = new Thread() {
public void run() {
DroolsLoadTest test = new DroolsLoadTest();
test.runTest();
try {
Thread.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
threads[i] = testThread;
testThread.start();
}
}
}
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
