On Jan 15, 2014, at 9:09 PM, Victor Hooi <[email protected]> wrote:
>
> I'm wondering if anybody knows of any web-scraping frameworks in Node.JS?
>
> Previously, there was node.io (https://github.com/chriso/node.io), however,
> the project was recently discontinued.
>
> Googling for Node.JS and web scraping, most of the guides online just talk
> about using requests and cheerio - it works, but you need to handle a whole
> bunch of things yourself (throttling, distributing jobs, configuration,
> managing jobs etc.).
>
There are a few modules (node-crawler, simple-crawler, etc) that might help
you. Ultimately you may have to wrap something around PhantomJS to deal with JS
modifications to the DOM (which can in turn be a bit of a pain since PhantomJS
for various reasons has to be run independently).
—ravi
> On the Python side, I know of Scrapy (https://github.com/scrapy/scrapy),
> which is using Twisted for asynchronicity
> On the Ruby side, Nokogiri (http://nokogiri.org/) is meant to be good,
> although I haven't dived into it much.
> Is there anything equivalent in the Node world? Or what approaches are people
> using to tackle this problem?
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
