We have developed a simple perl spider that takes a list of urls and downloads all the pages on the list, writing the downloaded pages to a directory.
Happy to share the script, just let me know. This script requires LWP::UserAgent, HTTP::Request, and HTTP::Response When a username/password is required to access a page, and it can be inserted in the individual url through the GET protocol, no problem. However with some .asp pages, the username/password are read from cookies, not from the location field. Has anybody figured a workaround for this situation? Is there another module and/or script that can be used to provide cookie-based password authentication? Thanks Mike Clark [EMAIL PROTECTED]
