I wrote spider for this but I haven't kept up with maintaining it: https://github.com/mikeal/spider
If you end up using it and have improvements to make i'll add you as a contributor. -Mikeal On Jan 20, 2014, at 10:44AM, Tim Killian <[email protected]> wrote: > Someone has already mentioned using cheerio, and I second that. I built a > basic web crawler/scraper using nothing but the requests and cheerio > libraries and it worked great. If you already know JQuery you already know > how to use Cheerio, which was also a big plus for me. > > On Wednesday, January 15, 2014 6:09:48 PM UTC-8, Victor Hooi wrote: > Hi, > > I'm wondering if anybody knows of any web-scraping frameworks in Node.JS? > > Previously, there was node.io (https://github.com/chriso/node.io), however, > the project was recently discontinued. > > Googling for Node.JS and web scraping, most of the guides online just talk > about using requests and cheerio - it works, but you need to handle a whole > bunch of things yourself (throttling, distributing jobs, configuration, > managing jobs etc.). > > On the Python side, I know of Scrapy (https://github.com/scrapy/scrapy), > which is using Twisted for asynchronicity > > On the Ruby side, Nokogiri (http://nokogiri.org/) is meant to be good, > although I haven't dived into it much. > > Is there anything equivalent in the Node world? Or what approaches are people > using to tackle this problem? > > Cheers, > Victor > > -- > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
