Hi there, I'm the author of sitescooper, http://sitescooper.org/ , a GPL'ed perl app for UNIX, Windows and Macs which snarfs the important bits of news sites (just the text, skipping ads etc.), and converts the HTML into other formats for reading offline on e.g. a Palm handheld. Thanks for LWP... I used to use a homebrew HTTP client lib, but so many sites used so many features of HTTP (cookies, redirects, etc.) it just became obvious that LWP was the way to go, and I'm glad I took that route. Firstly, any chance of a listing on http://www.linpro.no/lwp/ in the Applications section? cheers ;) Secondly, I've run into a wee problem (finally, after months of using LWP, about time too ;). I subclass LWP::UserAgent and provide an impl of redirect_ok(). This is necessary to handle rewriting redirected links in the output HTML. Let's say I go to www.nytimes.com and post a login request to their auth CGI. It will respond with: (some headers left out for clarity) HTTP/1.0 302 Found Set-cookie: NYT-S=1848Snkvy1Q3sLApigKgArhs6g3X6hAB02PkYX0002FlYQWCcCQN5aiV88F/w3O90nieL4ZhdXbf9oOG./AKE9/AYhfJCICzIn; expires=Saturday, 30-Oct-2010 22:45:59 GMT; path=/; domain=.nytimes.com Set-cookie: RDB=C8020000000000494501003C4519270101000000000002; expires=Tuesday, 30-Jan-2001 22:45:59 GMT; path=/; domain=.nytimes.com Location: http://www.nytimes.com/2000/11/01/politics/01CND-SPEND.html Obviously I need to set those cookies to proceed, on both the redirected request, and further requests to that site. Since LWP::UserAgent::request() doesn't pass the $response to redirect_ok(), that method cannot handle the cookies. I can't see any other way to do it at the moment. So, any chance you could do one of the following: 1. pass $response to redirect_ok(), so user code can implement this 2. provide a version of LWP::UserAgent::request() that takes a cookie_jar argument, and call $cookie_jar->extract_cookies($response); and $cookie_jar->add_cookie_header($referral); for redirects. I guess 1 is the preferred option from the API preservation point of view, although most users IMHO would expect request() to handle cookies as in 2. BTW -- if 2 is preferred -- be sure to check $referral_uri is a HTTP URL, HTTP::Cookies dies on file: URLs due to a call to URI::port(). (in case you're curious: what I'm currently doing now in sitescooper is implementing my own version of request() which does 2, and is used if LWP::Version() >= 5.47.) Again, thx for LWP, it's saved me a ton of bother. Looking forward to LWPng ;) --j.
