Re: [rules-users] Drools 5.3 partitioned rule base
I have been reevaluating our use cases and have decided to make a couple modifications: 1.Remove jBPM and Ruleflow groups. While it certainly fit a need I have derived a new way to do the same thing. 2. After reviewing our current ruleset I decided to modify what the rules actually do. At the end of the day the rules are written to infer certain things about the order. With this in mind I have decided to let the rules generate the billing meta-data about the Order(i.e I am may be eligible for charge A and for cap B) and have post processing which will use this information accordingly. The post processing is where the order of the Orders will matter and I am currently thinking of a couple ways to handle this. 3. Partitioning the data by account will allow us to multithread easily when running rules. However rather then have 1 session per account I was thinking about having sessions that represent a group of accounts. 4. With 3 in mind I am now trying to determine the best way to do this. My thinking atm is to create x sessions(for this example let's say 5), each associated with a thread via fireUntilHalt(). Since orders are streamed in by another thread reading from the DB my thinking is this is the most efficient way. However these leads me to a couple questions. How would I determine that an order has been processes through the ruleset and is ready for post processing(with fireAllRules I know that once the rules have fired I can retract the Order and move it to post processing and perhaps this is the answer that will be provided). With fireUntilHalt I thought about raising an event which I would use CDI to listen for and then retract the Order and pass it along for post processing. Is this a good approach? I am open to any suggestion here. 5. After watching a video posted by salaboy(http://vimeo.com/27209589) I am looking into some of the technologies he used. Particularly Kryo. The idea would be that if an Order was eligible for certain post processing it would be serialized to disc with a ranking(i.e key). Another part of the process would then read one at a time and process them in order. Any thoughts around this(I know this is not a drools question but I figured I would put it out there anyway). Thanks Again -Mike laune wrote A few random observations. A recent experiment has shown that insert takes 0.82 seconds for 10 facts with one very simple rule with a CPU G620 @ 2.60GHz and: java version 1.7.0_02 Java(TM) SE Runtime Environment (build 1.7.0_02-b13) Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode) There is, of course, no upper limit for insert as this depends on the number and complexity of the rules. And if you apply the jBPM Process paradigm, I'm not prepared to make any guesses. (It stands to reason that this will incur more overhead than straightforward processing in the Expert engine.) Estimating 1K per order, I'd say that i/o time for 150M orders is not to be neglected. I'd investigate batch (but not bulk) insertion of quantities, so that inserts and fire calls remain in good proportion. Also, balancing i/o threads and processing threads in relation to available CPUs shouldn't be different for an application using Drools. I've been re-reading some of this thread, and the requirement of ordered processing is going to be very difficult if the order this to be maintained over 150M facts. I mentioned dynamic property, but many different salience levels for a large number of activations is not efficient, since this requires a sequential search. -W On 3 March 2012 21:30, gboro54 lt;gboro54@gt; wrote: My previous statement was poorly written(that is what I get for trying to respond via my cell phone ). I agree that the Order fact is simple enough to have a rule retract it once processing is completed. As far as the threading, this is an ejb application being deployed on JBoss 7.1. Because of this I was hoping to use @Asyn and allow the container to really manage the threads. This method would simply request the cached session insert the Order fact, and start the jBPM process to process the Order. Perhaps this part of the design is still flawed as I have only recently been thinking about this and I would welcome all suggestions. Additionally is there any documentation on utilizing drools in a high throughput environment(i.e white papers,etc.) This billing project is the first of many potential projects we are looking to utilize drools for and I would love to share some more documentation on this(as well as read on myself). gboro54 wrote Agreed that I can have a rule to retract the orderi was just thinking if the overhead of a reevaluation but can live with that...As far as the threading...this logic is going to be deployed on a jboss instance using ejb so I was thinking about, using the jboss thread pool...so when I say new thread it may not, be new put, pulled from a pooldoes that
Re: [rules-users] Drools 5.3 partitioned rule base
Many of the systems use large amounts of data or large numbers of rules by applying indices inside rules or by processing transactions that could be batched - which really increases the number of rules or that that are seen. One of my customers used indices in rules creating rulesets of ~ 200,000 rules. With analysis the number of rules was reduced to ~ 10 + some data structures. If account processing transactions can be grouped together then the number of transactions may be dramatically reduced. The approach of using a thinner client with seperate sessions on seperate processes using partitioned data is a good idea. If you don't have an intuititve key then you may want processes to processes chunks of queued data as processes are ready. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3804819.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
A few random observations. A recent experiment has shown that insert takes 0.82 seconds for 10 facts with one very simple rule with a CPU G620 @ 2.60GHz and: java version 1.7.0_02 Java(TM) SE Runtime Environment (build 1.7.0_02-b13) Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode) There is, of course, no upper limit for insert as this depends on the number and complexity of the rules. And if you apply the jBPM Process paradigm, I'm not prepared to make any guesses. (It stands to reason that this will incur more overhead than straightforward processing in the Expert engine.) Estimating 1K per order, I'd say that i/o time for 150M orders is not to be neglected. I'd investigate batch (but not bulk) insertion of quantities, so that inserts and fire calls remain in good proportion. Also, balancing i/o threads and processing threads in relation to available CPUs shouldn't be different for an application using Drools. I've been re-reading some of this thread, and the requirement of ordered processing is going to be very difficult if the order this to be maintained over 150M facts. I mentioned dynamic property, but many different salience levels for a large number of activations is not efficient, since this requires a sequential search. -W On 3 March 2012 21:30, gboro54 gbor...@gmail.com wrote: My previous statement was poorly written(that is what I get for trying to respond via my cell phone ). I agree that the Order fact is simple enough to have a rule retract it once processing is completed. As far as the threading, this is an ejb application being deployed on JBoss 7.1. Because of this I was hoping to use @Asyn and allow the container to really manage the threads. This method would simply request the cached session insert the Order fact, and start the jBPM process to process the Order. Perhaps this part of the design is still flawed as I have only recently been thinking about this and I would welcome all suggestions. Additionally is there any documentation on utilizing drools in a high throughput environment(i.e white papers,etc.) This billing project is the first of many potential projects we are looking to utilize drools for and I would love to share some more documentation on this(as well as read on myself). gboro54 wrote Agreed that I can have a rule to retract the orderi was just thinking if the overhead of a reevaluation but can live with that...As far as the threading...this logic is going to be deployed on a jboss instance using ejb so I was thinking about, using the jboss thread pool...so when I say new thread it may not, be new put, pulled from a pooldoes that sound reasonable? laune wrote Starting a new thread (if that's what you mean by process) for reusing an existing session to process another order is likely to create more overhead and it'll just make threads compete for this resources. Multiple threads each dedicated to a single session object might be a better way to go. I don't see any benefit to make simple order facts into events just for the sake of making them expire automatically. There ought to be a well defined state (or states) when processed orders are retracted by some rule. -W On 3 March 2012 14:11, gboro54 lt;gboro54@gt; wrote: Mark, I really appreciate the help and think I have come to a solution. Let me know if this sounds reasonable. 1. We will continue to use jBPM to coordinate the rules and business logic that need to occur in calculating charges for orders. However we will work the process to only work on a per order level. 2. All orders are associated with accounts. We will keep one knowledgebase as the rule sets are the same and we will partition sessions by accounts. The flow will go as follows: a. If the session exists insert the order, start a new process instance and fire all rules b. If the session has not been created: create the session, insert all reference data that will be used by all orders in executions of the rule set, insert the order, start a process, and fire all rules. This session is then cached(via a hashmap more then likely) c. This process will be invoked asyn from the main thread, allowing us to control the multithreading 3. Orders will be treated as an event and will be set to expire in a certain amount of time, allowing us to mange the memory footprint of each session. Does this sound reasonable based on what you know of our usecase? Additionally with expiring Orders does this cause a reevaluation of the rules when the expiration occurs? The only other question I have is does the expiration clock start when no more activation's get created for the given event? Thanks again. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html Sent from the Drools: User forum mailing list
Re: [rules-users] Drools 5.3 partitioned rule base
On 02/03/2012 18:14, gboro54 wrote: How does MultithreadEvaluationOption work on a kBase? It is not explained really clearly anywhere. I know if I insert single objects only this works but as soon as two more of an object goes in I begin getting NPE... The feature is experimental and currently has problems, it's not recommended you use it. Mark Mark Proctor wrote Drools doesn't partition. You will need to build an external framework that partitions your data, and then assign a ksession per parition. Whether it's a ksession or kbase per partition depends on whether you can use the same rules per partition or not. Mark On 02/03/2012 17:33, gboro54 wrote: Could someone explain a little more about how Drools does a partition. I believe I understand Mark's idea, I just need to flush it out some more -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794163.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794296.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
On 02/03/2012 17:52, gboro54 wrote: Would you partition the KnowledgeBase or the Sessions? Additional would each Order being evaluated create a new session? It depends on your problem and your data. Mark gboro54 wrote Could you elaborate on that a little more Mark? Mark Proctor wrote Parition yourself ahead of time. Choose a key that is a true partition of your data, chose the number of partitions, create a kbase per partition and then hash on your inserted data to get the target kbase. Mark On 02/03/2012 16:23, gboro54 wrote: We are writing a billing system using Drools to evaluate orders placed during the day(this is a month to date process which will run nightly and we will be bringing a real-time solution online later this year after we rewrite the existing). Base fees of these orders can happen in parallel and in no way affect one another, however we have price caps which do depend on the order in which the cap is applied to the order(for certain conditions on an order a surcharge may be created if a cap is applied). My past experience will Drools has been it is quicker to do as much evaluation up front rather then loop of a list of Orders and fire one at a time. However my experience in running Drools with this load is limited(by the end of the month we will have to process 15 million orders). I am open to suggestions on the best way to do this. Additionally orders are reprocessed each night as orders from a current day may affect the pricing of an order from previous days(i.e a tier rate may apply, etc.) On a side note I found that the issue is a NPE exception one of my rules which only occurs if I partition the rule base(I am not sure why this would make a difference). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793979.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794233.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Mark, I really appreciate the help and think I have come to a solution. Let me know if this sounds reasonable. 1. We will continue to use jBPM to coordinate the rules and business logic that need to occur in calculating charges for orders. However we will work the process to only work on a per order level. 2. All orders are associated with accounts. We will keep one knowledgebase as the rule sets are the same and we will partition sessions by accounts. The flow will go as follows: a. If the session exists insert the order, start a new process instance and fire all rules b. If the session has not been created: create the session, insert all reference data that will be used by all orders in executions of the rule set, insert the order, start a process, and fire all rules. This session is then cached(via a hashmap more then likely) c. This process will be invoked asyn from the main thread, allowing us to control the multithreading 3. Orders will be treated as an event and will be set to expire in a certain amount of time, allowing us to mange the memory footprint of each session. Does this sound reasonable based on what you know of our usecase? Additionally with expiring Orders does this cause a reevaluation of the rules when the expiration occurs? The only other question I have is does the expiration clock start when no more activation's get created for the given event? Thanks again. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Starting a new thread (if that's what you mean by process) for reusing an existing session to process another order is likely to create more overhead and it'll just make threads compete for this resources. Multiple threads each dedicated to a single session object might be a better way to go. I don't see any benefit to make simple order facts into events just for the sake of making them expire automatically. There ought to be a well defined state (or states) when processed orders are retracted by some rule. -W On 3 March 2012 14:11, gboro54 gbor...@gmail.com wrote: Mark, I really appreciate the help and think I have come to a solution. Let me know if this sounds reasonable. 1. We will continue to use jBPM to coordinate the rules and business logic that need to occur in calculating charges for orders. However we will work the process to only work on a per order level. 2. All orders are associated with accounts. We will keep one knowledgebase as the rule sets are the same and we will partition sessions by accounts. The flow will go as follows: a. If the session exists insert the order, start a new process instance and fire all rules b. If the session has not been created: create the session, insert all reference data that will be used by all orders in executions of the rule set, insert the order, start a process, and fire all rules. This session is then cached(via a hashmap more then likely) c. This process will be invoked asyn from the main thread, allowing us to control the multithreading 3. Orders will be treated as an event and will be set to expire in a certain amount of time, allowing us to mange the memory footprint of each session. Does this sound reasonable based on what you know of our usecase? Additionally with expiring Orders does this cause a reevaluation of the rules when the expiration occurs? The only other question I have is does the expiration clock start when no more activation's get created for the given event? Thanks again. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Agreed that I can have a rule to retract the orderi was just thinking if the overhead of a reevaluation but can live with that...As far as the threading...this logic is going to be deployed on a jboss instance using ejb so I was thinking about, using the jboss thread pool...so when I say new thread it may not, be new put, pulled from a pooldoes that sound reasonable? laune wrote Starting a new thread (if that's what you mean by process) for reusing an existing session to process another order is likely to create more overhead and it'll just make threads compete for this resources. Multiple threads each dedicated to a single session object might be a better way to go. I don't see any benefit to make simple order facts into events just for the sake of making them expire automatically. There ought to be a well defined state (or states) when processed orders are retracted by some rule. -W On 3 March 2012 14:11, gboro54 lt;gboro54@gt; wrote: Mark, I really appreciate the help and think I have come to a solution. Let me know if this sounds reasonable. 1. We will continue to use jBPM to coordinate the rules and business logic that need to occur in calculating charges for orders. However we will work the process to only work on a per order level. 2. All orders are associated with accounts. We will keep one knowledgebase as the rule sets are the same and we will partition sessions by accounts. The flow will go as follows: a. If the session exists insert the order, start a new process instance and fire all rules b. If the session has not been created: create the session, insert all reference data that will be used by all orders in executions of the rule set, insert the order, start a process, and fire all rules. This session is then cached(via a hashmap more then likely) c. This process will be invoked asyn from the main thread, allowing us to control the multithreading 3. Orders will be treated as an event and will be set to expire in a certain amount of time, allowing us to mange the memory footprint of each session. Does this sound reasonable based on what you know of our usecase? Additionally with expiring Orders does this cause a reevaluation of the rules when the expiration occurs? The only other question I have is does the expiration clock start when no more activation's get created for the given event? Thanks again. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3796613.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
My previous statement was poorly written(that is what I get for trying to respond via my cell phone ). I agree that the Order fact is simple enough to have a rule retract it once processing is completed. As far as the threading, this is an ejb application being deployed on JBoss 7.1. Because of this I was hoping to use @Asyn and allow the container to really manage the threads. This method would simply request the cached session insert the Order fact, and start the jBPM process to process the Order. Perhaps this part of the design is still flawed as I have only recently been thinking about this and I would welcome all suggestions. Additionally is there any documentation on utilizing drools in a high throughput environment(i.e white papers,etc.) This billing project is the first of many potential projects we are looking to utilize drools for and I would love to share some more documentation on this(as well as read on myself). gboro54 wrote Agreed that I can have a rule to retract the orderi was just thinking if the overhead of a reevaluation but can live with that...As far as the threading...this logic is going to be deployed on a jboss instance using ejb so I was thinking about, using the jboss thread pool...so when I say new thread it may not, be new put, pulled from a pooldoes that sound reasonable? laune wrote Starting a new thread (if that's what you mean by process) for reusing an existing session to process another order is likely to create more overhead and it'll just make threads compete for this resources. Multiple threads each dedicated to a single session object might be a better way to go. I don't see any benefit to make simple order facts into events just for the sake of making them expire automatically. There ought to be a well defined state (or states) when processed orders are retracted by some rule. -W On 3 March 2012 14:11, gboro54 lt;gboro54@gt; wrote: Mark, I really appreciate the help and think I have come to a solution. Let me know if this sounds reasonable. 1. We will continue to use jBPM to coordinate the rules and business logic that need to occur in calculating charges for orders. However we will work the process to only work on a per order level. 2. All orders are associated with accounts. We will keep one knowledgebase as the rule sets are the same and we will partition sessions by accounts. The flow will go as follows: a. If the session exists insert the order, start a new process instance and fire all rules b. If the session has not been created: create the session, insert all reference data that will be used by all orders in executions of the rule set, insert the order, start a process, and fire all rules. This session is then cached(via a hashmap more then likely) c. This process will be invoked asyn from the main thread, allowing us to control the multithreading 3. Orders will be treated as an event and will be set to expire in a certain amount of time, allowing us to mange the memory footprint of each session. Does this sound reasonable based on what you know of our usecase? Additionally with expiring Orders does this cause a reevaluation of the rules when the expiration occurs? The only other question I have is does the expiration clock start when no more activation's get created for the given event? Thanks again. -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3796689.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
[rules-users] Drools 5.3 partitioned rule base
I am trying to run a partitioned rule base as we have a need to evaluate a large number of facts. We have around 160 rules which are coordinated using jBPM rule-flow groups(intermingled with some lightweight business logic). The business logic in the process executed quickly(80ms for 10,000 facts) but the rules take sometime to evaluate(thus why I thought a partitioned rule base would work). However when I try to run it I get the stack trace below. Do I need to do something to the session as well? java.lang.IllegalStateException: Error launching test com.nasdaq.backoffice.billing.process.PhlxCoreProcessTest public void com.nasdaq.backoffice.billing.process.PhlxCoreProcessTest.testPhlxCoreProcess() at org.jboss.arquillian.protocol.servlet.ServletMethodExecutor.invoke(ServletMethodExecutor.java:122) at org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:120) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111)
Re: [rules-users] Drools 5.3 partitioned rule base
After investagation I found the following: 10:29:45,421 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(SingleObjectSinkAdapter.java:59) 10:29:45,423 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:134) 10:29:45,424 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.PartitionTaskManager$FactAssertAction.execute(PartitionTaskManager.java:272) 10:29:45,424 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.PartitionTaskManager$PartitionTask.run(PartitionTaskManager.java:116) 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at org.drools.concurrent.ExternalExecutorService$ObservableRunnable.run(ExternalExecutorService.java:443) 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) 10:29:45,423 INFO [stdout] (pool-38-thread-1) SURCHARGES created 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:29:45,426 ERROR [stderr] (pool-38-thread-2) at java.lang.Thread.run(Unknown Source) Any thoughts? -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793788.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Why are you trying with that partitioning approach? By the why I notice that most of the time when people uses ruleflow-groups they didn't realize that all the evaluations are done at insertion time, not when the process reaches the rule flow group, is that you case? Doing consulting around these topics most of the clients doesn't need to use rule flow groups and they just need some control facts to speed up the processing times. Cheers On Fri, Mar 2, 2012 at 3:33 PM, gboro54 gbor...@gmail.com wrote: After investagation I found the following: 10:29:45,421 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(SingleObjectSinkAdapter.java:59) 10:29:45,423 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:134) 10:29:45,424 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.PartitionTaskManager$FactAssertAction.execute(PartitionTaskManager.java:272) 10:29:45,424 ERROR [stderr] (pool-38-thread-2) at org.drools.reteoo.PartitionTaskManager$PartitionTask.run(PartitionTaskManager.java:116) 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at org.drools.concurrent.ExternalExecutorService$ObservableRunnable.run(ExternalExecutorService.java:443) 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) 10:29:45,423 INFO [stdout] (pool-38-thread-1) SURCHARGES created 10:29:45,425 ERROR [stderr] (pool-38-thread-2) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 10:29:45,426 ERROR [stderr] (pool-38-thread-2) at java.lang.Thread.run(Unknown Source) Any thoughts? -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793788.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- - MyJourney @ http://salaboy.wordpress.com - Co-Founder @ http://www.jugargentina.org - Co-Founder @ http://www.jbug.com.ar - Salatino Salaboy Mauricio - ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
I understand that all the evaluations are happening at insertion time. The main reason for the groups is to intermix logic needed to execute outside of rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying to partition because I need to be able to get a high level of messages through these rules(a min of 1000/sec). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793879.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
I understand that all the evaluations are happening at insertion time. The main reason for the groups is to intermix logic needed to execute outside of rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying to partition because I need to be able to get a high level of messages through these rules(a min of 1000/sec). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793880.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
So the question is.. for processing those messages, you can set up multiple sessions and evaluate each message separately? or do you need to have all the messages in the same session? Cheers On Fri, Mar 2, 2012 at 3:58 PM, gboro54 gbor...@gmail.com wrote: I understand that all the evaluations are happening at insertion time. The main reason for the groups is to intermix logic needed to execute outside of rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying to partition because I need to be able to get a high level of messages through these rules(a min of 1000/sec). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793879.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- - MyJourney @ http://salaboy.wordpress.com - Co-Founder @ http://www.jugargentina.org - Co-Founder @ http://www.jbug.com.ar - Salatino Salaboy Mauricio - ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Executions need to happen in an order based on a data and a message id(at least at one part of the evaluation). Because of this my plan was to use dynamic salience and insert as many of the facts as I can at a time -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793941.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
But are the facts related with each other? do you need to analyze them all together inside the same session? Can you share a little bit more about your use cases? What do you mean with Execution execution of what? What do you mean with evaluation? Cheers On Fri, Mar 2, 2012 at 4:12 PM, gboro54 gbor...@gmail.com wrote: Executions need to happen in an order based on a data and a message id(at least at one part of the evaluation). Because of this my plan was to use dynamic salience and insert as many of the facts as I can at a time -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793941.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -- - MyJourney @ http://salaboy.wordpress.com - Co-Founder @ http://www.jugargentina.org - Co-Founder @ http://www.jbug.com.ar - Salatino Salaboy Mauricio - ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
We are writing a billing system using Drools to evaluate orders placed during the day(this is a month to date process which will run nightly and we will be bringing a real-time solution online later this year after we rewrite the existing). Base fees of these orders can happen in parallel and in no way affect one another, however we have price caps which do depend on the order in which the cap is applied to the order(for certain conditions on an order a surcharge may be created if a cap is applied). My past experience will Drools has been it is quicker to do as much evaluation up front rather then loop of a list of Orders and fire one at a time. However my experience in running Drools with this load is limited(by the end of the month we will have to process 15 million orders). I am open to suggestions on the best way to do this. Additionally orders are reprocessed each night as orders from a current day may affect the pricing of an order from previous days(i.e a tier rate may apply, etc.) On a side note I found that the issue is a NPE exception one of my rules which only occurs if I partition the rule base(I am not sure why this would make a difference). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793979.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Parition yourself ahead of time. Choose a key that is a true partition of your data, chose the number of partitions, create a kbase per partition and then hash on your inserted data to get the target kbase. Mark On 02/03/2012 16:23, gboro54 wrote: We are writing a billing system using Drools to evaluate orders placed during the day(this is a month to date process which will run nightly and we will be bringing a real-time solution online later this year after we rewrite the existing). Base fees of these orders can happen in parallel and in no way affect one another, however we have price caps which do depend on the order in which the cap is applied to the order(for certain conditions on an order a surcharge may be created if a cap is applied). My past experience will Drools has been it is quicker to do as much evaluation up front rather then loop of a list of Orders and fire one at a time. However my experience in running Drools with this load is limited(by the end of the month we will have to process 15 million orders). I am open to suggestions on the best way to do this. Additionally orders are reprocessed each night as orders from a current day may affect the pricing of an order from previous days(i.e a tier rate may apply, etc.) On a side note I found that the issue is a NPE exception one of my rules which only occurs if I partition the rule base(I am not sure why this would make a difference). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793979.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Could you elaborate on that a little more Mark? Mark Proctor wrote Parition yourself ahead of time. Choose a key that is a true partition of your data, chose the number of partitions, create a kbase per partition and then hash on your inserted data to get the target kbase. Mark On 02/03/2012 16:23, gboro54 wrote: We are writing a billing system using Drools to evaluate orders placed during the day(this is a month to date process which will run nightly and we will be bringing a real-time solution online later this year after we rewrite the existing). Base fees of these orders can happen in parallel and in no way affect one another, however we have price caps which do depend on the order in which the cap is applied to the order(for certain conditions on an order a surcharge may be created if a cap is applied). My past experience will Drools has been it is quicker to do as much evaluation up front rather then loop of a list of Orders and fire one at a time. However my experience in running Drools with this load is limited(by the end of the month we will have to process 15 million orders). I am open to suggestions on the best way to do this. Additionally orders are reprocessed each night as orders from a current day may affect the pricing of an order from previous days(i.e a tier rate may apply, etc.) On a side note I found that the issue is a NPE exception one of my rules which only occurs if I partition the rule base(I am not sure why this would make a difference). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793979.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794015.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Could someone explain a little more about how Drools does a partition. I believe I understand Mark's idea, I just need to flush it out some more -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794163.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Would you partition the KnowledgeBase or the Sessions? Additional would each Order being evaluated create a new session? gboro54 wrote Could you elaborate on that a little more Mark? Mark Proctor wrote Parition yourself ahead of time. Choose a key that is a true partition of your data, chose the number of partitions, create a kbase per partition and then hash on your inserted data to get the target kbase. Mark On 02/03/2012 16:23, gboro54 wrote: We are writing a billing system using Drools to evaluate orders placed during the day(this is a month to date process which will run nightly and we will be bringing a real-time solution online later this year after we rewrite the existing). Base fees of these orders can happen in parallel and in no way affect one another, however we have price caps which do depend on the order in which the cap is applied to the order(for certain conditions on an order a surcharge may be created if a cap is applied). My past experience will Drools has been it is quicker to do as much evaluation up front rather then loop of a list of Orders and fire one at a time. However my experience in running Drools with this load is limited(by the end of the month we will have to process 15 million orders). I am open to suggestions on the best way to do this. Additionally orders are reprocessed each night as orders from a current day may affect the pricing of an order from previous days(i.e a tier rate may apply, etc.) On a side note I found that the issue is a NPE exception one of my rules which only occurs if I partition the rule base(I am not sure why this would make a difference). -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3793979.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794233.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Drools doesn't partition. You will need to build an external framework that partitions your data, and then assign a ksession per parition. Whether it's a ksession or kbase per partition depends on whether you can use the same rules per partition or not. Mark On 02/03/2012 17:33, gboro54 wrote: Could someone explain a little more about how Drools does a partition. I believe I understand Mark's idea, I just need to flush it out some more -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794163.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
How does MultithreadEvaluationOption work on a kBase? It is not explained really clearly anywhere. I know if I insert single objects only this works but as soon as two more of an object goes in I begin getting NPE... Mark Proctor wrote Drools doesn't partition. You will need to build an external framework that partitions your data, and then assign a ksession per parition. Whether it's a ksession or kbase per partition depends on whether you can use the same rules per partition or not. Mark On 02/03/2012 17:33, gboro54 wrote: Could someone explain a little more about how Drools does a partition. I believe I understand Mark's idea, I just need to flush it out some more -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794163.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users ___ rules-users mailing list rules-users@.jboss https://lists.jboss.org/mailman/listinfo/rules-users -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794296.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
Additionally when does partitioning by data cause performance to be compromised? It has always been in my experience and to my knowledge that building a complete session with facts is cheaper then iterating a list of facts, creating a session, and firing. Perhaps I am mistaken? Additionally is there documentation anywhere talking about utilizing drools in a high volume environment? -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794308.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
Re: [rules-users] Drools 5.3 partitioned rule base
I was thinking we could do separate sessions then queue up the results in order and process themI also like Mark's idea of partitioning -- View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3794439.html Sent from the Drools: User forum mailing list archive at Nabble.com. ___ rules-users mailing list rules-users@lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users
