Hi Richard, I would really like to help you, but I don't have the spare cycles to help in depth with application design in this forum. If you have started with work pulling (which is also pushing to idle workers) it might be a good idea to try that for this as well. It gives you the backpressure (not overloading workers) part that is so important.
You have probably already read Ryan's blog post, otherwise that is a great read: http://blog.goconspire.com/post/64130417462/akka-at-conspire-part-1-how-we-built-our-backend-on Make sure you make it to the end of the series ("The Importance of Pulling") Cheers, Patrik On Thu, Jul 3, 2014 at 8:51 PM, Richard Rodseth <[email protected]> wrote: > For some reason I'm really struggling to find the right design here. I > feel OK about my "channel job" actor which creates as children a job > tracker, a worker pull master, and some workers. It builds a work list of > time periods (currently assumed to be from one channel) which it sends to > the child worker-pull master. The tracker sends a job complete message to > the job actor and the job actor stops itself. So I guess the job actor it's > a bit like a per-request actor where the request is not a web request, but > a request to process a channel. > > But now I have millions of channels to process. I can paginate them or > organize them in other ways, like location. I'd like to start as many > channel jobs in parallel as possible and get a good distribution across > location. How can I throttle this job creation or is there some other > pattern I'm missing? Should I just turn the millions of channels into > millions of messages to a router and hope I can beef up the number of > routees? Should I re-apply worker pull where each worker works on a chunk > of channels? If so, where do I get the Future in overide def > doWork():Future[_] from? By ask?ing the list of per-channel jobs their > status? > > Sorry if I'm being dense. > > > On Tue, Jul 1, 2014 at 6:08 PM, Richard Rodseth <[email protected]> > wrote: > >> I've had some success using the worker pull pattern to process time >> series data in a particular "channel". A channel job actor builds the queue >> of time chunks for a work pull master and workers work on chunks of time. >> >> Now I wish to process multiple (possibly millions of) channels in >> parallel and am wondering whether to a) repeat the pattern, treating a page >> of channels as the work queue item, where the page workers would spawn a >> channel job per channel, or b) use something else like routers or balancing >> dispatcher. Any tips? > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
