> From: discuss-boun...@lists.lopsa.org [mailto:discuss- > boun...@lists.lopsa.org] On Behalf Of Ski Kacoroski > > So do any of you have any great ideas, wonderful software, etc that can > scrape a website on a regular basis so I could at least have provided > the content back to the teacher. I will need to get the pages > (including pages buried behind javascript and ajax buttons and menus) > along with attached files).
If some of the relevant content is behind javascript/ajax, etc, then your usual crawlers (curl,wget,etc) aren't going to cut the butter for you... You'll probably need a "real" web scraping solution, like using selenium and writing your own custom scraping app. I've used selenium in C#, but I'm pretty sure it's also available in java, python, and other languages. And there are probably some other libraries out there that compete against Se. _______________________________________________ Discuss mailing list Discuss@lists.lopsa.org https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss This list provided by the League of Professional System Administrators http://lopsa.org/
