Hey Julien/Adam, There was a great thread about a similar situation about 10 days ago. Check it out here: http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001681.html
Julien, you definitely don't want a worker for each feed, and you'll want to use thread_pool.defer, which will allow you to concurrently process as many feeds as you want (or as many as your system can handle). From what you've said, it sounds like you'll only need one worker coded up, but probably set multiple periodic timers (e.g one of hourly parsing of high-priority feeds, one for nightlies, etc). The method you specify in the periodic timer should use thread_pool.defer to handle processing of multiple feeds at a time -- there's no reason to do them sequentially. stevie On Wed, Apr 23, 2008 at 10:30 AM, Julien Genestoux <[EMAIL PROTECTED]> wrote: > Thanks Adam, > > That sounded weird to me as well to have one worker for each feed... > However, if I only have one worker, that also means that I am parsing > one feed only at any moment. An option, is maybe to have a few workers > (denpending on the number of feeds) that parse feeds concurrently? > > If I only have one worker, according to you what should be the > winnning strategy to choose the "right" parse to feed? Obviously some > feeds need to be parsed one every few minutes, while some other might > no need to be parse more than every hour... > > Any idea/tip on this? > > > > > > > On 4/23/08, Adam Williams <[EMAIL PROTECTED]> wrote: > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > I still have a few questions : shoud I have one worker for each feed > > > that is called periodically (add_periodic_timer) or rather one single > > > worker that calls every feed one by one? > > > > > > What is the best solution, perfomance-wise? > > > > > > Good question... I don't suppose I know exactly. I would start by > > processing all the feeds in one worker invocation - that is what I > > have done for sending an unknown amount of email. It just seems wrong > > to me to invoke a worker for one email at a time. > > > > The right answer likely lies in understanding the whole MasterWorker, > > Packet::Reactor/handler_instance.ask_work bits of the puzzle... > > > > > > adam > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > [email protected] > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > -- > Julien Genestoux > [EMAIL PROTECTED] > http://www.ouvre-boite.com > +1 (415) 254 7340 > +33 (0)8 70 44 76 29 > _______________________________________________ > > > Backgroundrb-devel mailing list > [email protected] > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > _______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
