For some reason I'm really struggling to find the right design here. I feel OK about my "channel job" actor which creates as children a job tracker, a worker pull master, and some workers. It builds a work list of time periods (currently assumed to be from one channel) which it sends to the child worker-pull master. The tracker sends a job complete message to the job actor and the job actor stops itself. So I guess the job actor it's a bit like a per-request actor where the request is not a web request, but a request to process a channel.
But now I have millions of channels to process. I can paginate them or organize them in other ways, like location. I'd like to start as many channel jobs in parallel as possible and get a good distribution across location. How can I throttle this job creation or is there some other pattern I'm missing? Should I just turn the millions of channels into millions of messages to a router and hope I can beef up the number of routees? Should I re-apply worker pull where each worker works on a chunk of channels? If so, where do I get the Future in overide def doWork():Future[_] from? By ask?ing the list of per-channel jobs their status? Sorry if I'm being dense. On Tue, Jul 1, 2014 at 6:08 PM, Richard Rodseth <[email protected]> wrote: > I've had some success using the worker pull pattern to process time series > data in a particular "channel". A channel job actor builds the queue of > time chunks for a work pull master and workers work on chunks of time. > > Now I wish to process multiple (possibly millions of) channels in parallel > and am wondering whether to a) repeat the pattern, treating a page of > channels as the work queue item, where the page workers would spawn a > channel job per channel, or b) use something else like routers or balancing > dispatcher. Any tips? -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
