I am happy to take any form of contribution you can offer. By adding additional worker endpoints I mean.
Load Balancer 1, 5 threads, deployed to MachineA 1 worker endpoint, configured to send to Machine1\queue1.readyforwork, 5 threads, deployed to NewMachineB 2 worker endpoint, configured to send to Machine1\queue1.readyforwork, 5 threads, deployed to NewMachineC Load balancing although completely *possible* to run on one machine, was designed to distribute load to multiple machines. You're not gaining any benefits from load balancing when there is only one worker sending ready for work messages to the load balancer. You would be better off in this case just having two endpoints without load balancing. On Tue, Nov 15, 2011 at 10:28 PM, Michael Lyons <[email protected]>wrote: > I've run EQATEC profiler against the code and when the load balancer > process is under load it it records no activity between snapshots > indicating it is sitting in RSB code. > > I'd be happy to spot profile RSB in my app and point out where the > high CPU is coming from but I'm assuming you already have a fair idea. > > What do you mean by adding additional worker endpoints? Can you point > me to an example. > > > On Nov 16, 3:40 pm, Corey Kaylor <[email protected]> wrote: > > I would try changing the thread counts on the consumers and the load > > balancer, and possibly add additional worker endpoint(s). > > > > Ayende in previous conversations has recommended thread counts that are > > equal to the number of cores on the machine. I have found that isn't > always > > a perfect recipe. So in our case we have run load tests and changing the > > configuration of threads for each machine. > > > > When changing the thread counts on each test run, try to observe which > > specific process is utilizing the most CPU. > > > > There may be places to optimize for sure, but it sounds to me like > threads > > are competing for priority. > > > > > > > > > > > > > > > > On Tue, Nov 15, 2011 at 9:24 PM, Michael Lyons <[email protected]> > wrote: > > > Yes you're correct, it's a staging environment where we do our testing > > > before releasing into production. > > > > > That's pretty much the situation. > > > > > Here are the xml configurations for the 2 load balancers: > > > > > <loadBalancer threadCount="5" > > > endpoint="msmq://localhost/notifier.loadbalancer" > > > readyForWorkEndpoint="msmq://localhost/ > > > notifier.loadbalancer.acceptingwork" > > > /> > > > > > <loadBalancer threadCount="5" > > > endpoint="msmq://localhost/processor.loadbalancer" > > > readyForWorkEndpoint="msmq://localhost/ > > > processor.loadbalancer.acceptingwork" > > > /> > > > > > Consumers xml configuration is: > > > > > <bus threadCount="20" > > > loadBalancerEndpoint="msmq://localhost/ > > > processor.loadbalancer.acceptingwork" > > > numberOfRetries="5" > > > endpoint="msmq://localhost/processor" > > > /> > > > > > <bus threadCount="20" > > > loadBalancerEndpoint="msmq://localhost/ > > > notifier.loadbalancer.acceptingwork" > > > numberOfRetries="5" > > > endpoint="msmq://localhost/notifier" > > > /> > > > > > On Nov 16, 3:13 pm, Corey Kaylor <[email protected]> wrote: > > > > To summarize your setup. > > > > > > Load Balancer 1, configured for messages belonging to NamespaceA, > with 5 > > > > threads, deployed to MachineA\queue1 > > > > 1 worker endpoint sending sending ready for work to > > > > MachineA\queue1.readyforwork, configured with 20 threads, deployed to > > > > MachineA > > > > > > Load Balancer 2, configured for messages belonging to NamespaceB, > with 5 > > > > threads, deployed to MachineA\queue2 > > > > 1 worker endpoint sending ready for work to > > > > MachineA\queue2.readyforwork, configured with 20 threads, deployed to > > > > MachineA > > > > > > I assumed by staging server that you mean staging environment that is > > > > configured similarly above but with different machine specs as you've > > > > stated. > > > > > > Is this correct? > > > > > > On Tue, Nov 15, 2011 at 8:12 PM, Michael Lyons <[email protected] > > > > > wrote: > > > > > The load balancers are configured with the readyForWorkEndpoint > > > > > attribute on the loadBalancer xml element. > > > > > > > System is a quad core 2.83Ghz core 2 duo, on the staging server > which > > > > > is running an older single core 2.8Ghz xeon (Dell 2650) with hyper > > > > > threading it sits at about 80% and in production it sits between > 40 to > > > > > 80% on a quad core 2.8Ghz xeon (Dell R210) where it is allocated 2 > > > > > cores > > > > > > > Forgot to mention that RSB is version 2.2 > > > > > > > On Nov 16, 1:17 pm, Corey Kaylor <[email protected]> wrote: > > > > > > Also, how many cores are on the load balancer machine? There > > > shouldn't be > > > > > > that much demand on the cpu, but having said that it really > depends > > > on > > > > > the > > > > > > circumstances and environment. > > > > > > > > On Tue, Nov 15, 2011 at 7:15 PM, Corey Kaylor <[email protected] > > > > > wrote: > > > > > > > Is each load balancer configured with a ready for work uri? > > > > > > > > > On Mon, Nov 14, 2011 at 5:06 PM, Michael Lyons < > > > [email protected] > > > > > >wrote: > > > > > > > > >> When using the load balancer with RSB I'm seeing the CPU runs > at > > > near > > > > > > >> 100% when the consumers are all busy which causes the > consumers > > > to run > > > > > > >> slower and be free less often. > > > > > > >> It can be simulated easily by setting up a load balancer with > no > > > > > > >> consumers listening to it and trying to send out some > messages to > > > the > > > > > > >> consumer. > > > > > > > > >> In my specific situation I have 2 load balancers with 5 > threads > > > each > > > > > > >> (each load balancer runs a separate queue with different > types of > > > > > > >> messages), there is a consumer waiting at the other end of > each > > > load > > > > > > >> balancer with another 20 threads each. If one of the load > > > balancers > > > > > gets > > > > > > >> congested then all consumers run slow. When I ran the load > > > balancer > > > > > without > > > > > > >> load it averaged ~200ms to process a message, once the load > > > balancer > > > > > was > > > > > > >> under load (achieved by queuing over 1000 messages) it > resulted > > > in an > > > > > > >> average time of ~1750ms, which results in the user waiting 8 > times > > > > > longer > > > > > > >> for their tasks to complete. > > > > > > > > >> Is there anyway around this? > > > > > > > > >> -- > > > > > > >> You received this message because you are subscribed to the > Google > > > > > Groups > > > > > > >> "Rhino Tools Dev" group. > > > > > > >> To view this discussion on the web visit > > > > > > >>https://groups.google.com/d/msg/rhino-tools-dev/-/PYwMBzLg7m4J > . > > > > > > >> To post to this group, send email to > > > [email protected] > > > > > . > > > > > > >> To unsubscribe from this group, send email to > > > > > > >> [email protected]. > > > > > > >> For more options, visit this group at > > > > > > >>http://groups.google.com/group/rhino-tools-dev?hl=en. > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Rhino Tools Dev" group. > > > > > To post to this group, send email to > [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/rhino-tools-dev?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Rhino Tools Dev" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]. > > > For more options, visit this group at > > >http://groups.google.com/group/rhino-tools-dev?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Rhino Tools Dev" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rhino-tools-dev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
