Sounds like you just want a node service that does the scraping, and provides an http api for adding tasks and querying what tasks are currently running. Then have a separate command-line app that can talk to the web service. And maybe also an express app that can talk to it via a browser. I use that pattern a lot, but haven't yet written an abstraction for it. An abstraction for easily banging out command-line apps with tab-complete, etc that just talk to web services would be a useful thingie though.
I'm not sure if there's a generic node-based job-processor out there, but you might like https://github.com/defunkt/resque. It's ruby, but each job could just be a system call to start a node process. Ted On Oct 5, 2012, at 4:22 AM, Guilherme Pim <[email protected]> wrote: > Okay, my bad! > > I am crawling some web pages to populate a database, and currently doing that > with Grunt (http://gruntjs.com/), which is a task-based 'build tool' (that > gets me confused, is it supposed to solely build?). > > So, I have created some scripts in the following manner: > > ========== Task file ========== > > module.exports = function (grunt) { > > grunt.registerTask('courses/update-info', 'Updates all courses info on > the database', ... ) > > } > > ============================ > > So then I may have a somewhat controlled access to these jobs from the > terminal, with ability to list all existing jobs and their descriptions. > Also, I may pass some parameters to each task, although it's not very > readable because they can't be associated with a name: > > grunt courses/update-info:4:5:2 (these will be passed as the tasks's function > arguments) > > I guess I could go about creating node scripts and using a parameters library > along with console colors and stuff, but I would rather find a framework, if > such exists. > > Did I make myself clearer at all? haha > > Thanks for you time, > Guilherme > > 2012/10/5 Nuno Job <[email protected]> > Can you share some more details on what you are trying to accomplish? > > You have talked about tools (both of which I am unaware of) and said "batch", > but apart from that there wasn't much more that could help anyone make an > informed suggestion. > > Let me know, I'll try to help :) > Nuno > > On Fri, Oct 5, 2012 at 11:08 AM, Guilherme Pim <[email protected]> wrote: > Hi guys, > > Having the need to synchronize some data with external sources, I have > written some tasks on Grunt to do so. I wonder if Grunt is actually the tool > to write batch jobs in, or there actually is some other alternative. > > I have come across Node.io, but I rather stick to Grunt in this case. Do you > guys know about some alternatives, please? > > Thanks! > Guilherme > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
