send-off, number of threads processor load
Hello all I have question about send-off, agents threads - I'm trying to use send-off to dispatch tasks, that stores some data into central agent. Tasks could be long running. I found following situation: - when I explicitly create threads and run tasks in them, I have more load on machine, that lead to bigger number of simultaneous tasks processed. Typically i see about 600% load on my machine with 8 logical processors (4cores with hyperthreading) - when I use send-off, I get less number of simultaneous connection to service, and load usually not bigger as 160% Questions are: how could I control how much treads are used in agent's queue? How can I improve payload on my machine, so Clojure will use all available resources? -- With best wishes, Alex Ott, MBA http://alexott.blogspot.com/ http://alexott.net http://alexott-ru.blogspot.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: send-off, number of threads processor load
On Tue, Feb 16, 2010 at 7:24 AM, Alex Ott alex...@gmail.com wrote: Hello all I have question about send-off, agents threads - I'm trying to use send-off to dispatch tasks, that stores some data into central agent. Tasks could be long running. I found following situation: - when I explicitly create threads and run tasks in them, I have more load on machine, that lead to bigger number of simultaneous tasks processed. Typically i see about 600% load on my machine with 8 logical processors (4cores with hyperthreading) - when I use send-off, I get less number of simultaneous connection to service, and load usually not bigger as 160% I don't think there's any limit on the number of threads spawned by send-off. ...except that no single agent will ever be allocated more than one thread at a time. Are you perhaps doing lots of send-offs to just one or two agents? --Chouser http://joyofclojure.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: send-off, number of threads processor load
Hello Chouser at Tue, 16 Feb 2010 12:02:17 -0500 wrote: C On Tue, Feb 16, 2010 at 7:24 AM, Alex Ott alex...@gmail.com wrote: Hello all I have question about send-off, agents threads - I'm trying to use send-off to dispatch tasks, that stores some data into central agent. Tasks could be long running. I found following situation: - when I explicitly create threads and run tasks in them, I have more load on machine, that lead to bigger number of simultaneous tasks processed. Typically i see about 600% load on my machine with 8 logical processors (4cores with hyperthreading) - when I use send-off, I get less number of simultaneous connection to service, and load usually not bigger as 160% C I don't think there's any limit on the number of threads spawned C by send-off. ...except that no single agent will ever be C allocated more than one thread at a time. Are you perhaps doing C lots of send-offs to just one or two agents? Yes, all send-off are sent to one agent, that accumulate information about different documents -- With best wishes, Alex Ott, MBA http://alexott.blogspot.com/http://alexott.net/ http://alexott-ru.blogspot.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: send-off, number of threads processor load
On 17 February 2010 06:18, Alex Ott alex...@gmail.com wrote: send-off to dispatch tasks, that stores some data into central agent. Tasks could be long running. Yes, all send-off are sent to one agent, that accumulate information about different documents As Chouser pointed out, agents execute one thread at a time such that their data access is synchronized. This is the coordination model they expose. You want multiple threads so you need a different coordination model. I would suggest using a ref as the central data store that can be accessed by multiple worker threads. A convenient way of spawning CPU limited worker threads is to use (submit-future) to be found at the end of this thread: http://groups.google.com/group/clojure/browse_thread/thread/51cea7e25fdfe653 or at github http://github.com/timothypratley/strive/blob/master/clj/timothypratley/extensions.clj However there are of course many different approaches to consider: * use pcalls if appropriate * use multiple agents * use futures or threads * do work on thread but coordinate with ref/agent/atom Regards, Tim. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
