A couple of random thoughts that might help. It sounds like you probably want to create a module that examines the feed, the last time new results were found, and time of day to determine when it should next check that feed or other feeds. Then you could use a module like https://github.com/mattpat/node-schedule to schedule the next check. An approach like this would work best as a long running process and logic that runs once at startup to schedule the first check for each feed. If you wanted to distribute the work across multiple workers and/or have saved scheduled jobs something like https://github.com/learnboost/kue might be worth looking at.
-- Daniel R. <[email protected]> [http://danielr.neophi.com/] On Fri, Mar 29, 2013 at 12:15 AM, Hady <[email protected]> wrote: > Hello everyone, > > I have few questions regarding a node application I'm trying to create and > optimize. > > Basically the application pulls down a set of rss/atom feeds > from different news websites, check for new articles and post them into a > facebook page or send some kind of notifications (I have a mongodb to keep > track of the ones that I have already gotten). I'm trying to optimize this > thing as much as I can, and do the minimum amount of rss > fetching through the day. > > The websites I'm interested in, do not update their news at the same > frequency some updates in every 1-2 hours and some takes more than that; so > I need separated updater for each website or group. I first used cronJob > module and set a different frequency for each updater; but then I realized > I can still lower the frequency at night time because there were no > updates; but I couldn't do that with cron job. > > Some of these websites cover the same kind of news, so if one updates, > it's highly expected that the other one will update shortly.. also they > tend to post bunch of news at the same time, so if I get an update I should > check back soon. > > What do you think of my current set up, and how can I improve it? > How would I be able run the updater with different frequency based on > custom conditions? are there any existing tools or modules? I dont want to > have to manage bunch of timeouts and callbacks.. > When running an updater, is it better to start a process and kill it when > it's done? or to have a single process running all time and uses setTimeout > to sleep? > > Thanks > -Hady > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
