On Wed, Feb 24, 2016 at 9:58 PM, Charles Curley <[email protected]> wrote: > I have been using Perl's www:mechanize to scrape a series of web pages. > > Unfortunately the web page now includes some Javascript, which > mechanize does not handle. Suggestions? > > All of my code is in shell script and perl, so I'd like to stick with > those. > > Suggestions?
You don't say what your overall intent is, but if it's testing (or several others), you won't regret the switch to Selenium/webdriver. There are lots of supported drivers (including phantomjs, htmlunit, and real browsers), and LOTS of add-on tools and services that are based on the webdriver API. The wire protocol is a W3C draft standard, and it works pretty well. I haven't used the perl bindings, but from what I understand, they're well maintained and widely used. If the app you're scraping is an AngularJS app, you really should look at protractor, even though that requires ditching perl. FWIW, Barry /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
