Mechanize Redirect
Have seen this issue addressed on this board in the past, but can't figure out exactly what I need to do. I'm using WWW::Mechanize and a webpage seems to be redirecting me but Mechanize doesn't seem to follow it. I've heard to Add the header 'Accept: text/html' but alas I don't know how to add the header, or what that means. I suspect the solution is simple belief - but alas I'm not coming up with it. Any help would be much appreciated.
Re: Mechanize - redirect problem
Hi Andy, Oops, sorry, I misquoted you. Whether FAQ, Cookbook or something new, I think this kind of practical hints on how to use mech would be tremendously valuable (also the reason I sent you some extra questions and their answers in response to your request). BTW - I don't think that cookbooks are real helpful. By the time they get to the users who need them, the websites have changed enough that they recipes don't work anymore. This was a big problem with the O'Reilly book. I bought it, but the examples I looked at didn't work. Cheers, Peter Andy Lester wrote: On Fri, Feb 25, 2005 at 10:25:35AM +0100, Peter Stevens ([EMAIL PROTECTED]) wrote: Andy - this really belongs in your tips tricks page. BTW - where is the page on the net? Do you mean the FAQ? I never said tips tricks. There's a Cookbook.pod and a FAQ.pod, both shipping with Mech. -- -- Peter Stevens Phone: +41 43 535 8517 www.MinuteWatcher.com Fax: +41 44 544 8392
Re: Mechanize - redirect problem
On Fri, Feb 25, 2005 at 05:46:44PM +0100, Peter Stevens ([EMAIL PROTECTED]) wrote: Oops, sorry, I misquoted you. Whether FAQ, Cookbook or something new, I think this kind of practical hints on how to use mech would be tremendously valuable (also the reason I sent you some extra questions and their answers in response to your request). I agree. Send me something and I'll drop it in. xoa -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Re: Mechanize - redirect problem
hi john It wants this header (or similar, but this is a minimal one): Accept: text/html i have added this header and it just works!!! thanks a LOT! Maybe mechanize should sent an Accept header by default? i think that would be a good idea for the text/html type. BTW, Martin: I debugged this by just looking at what Firefox sends. Get livehttpheaders. very handy firefox-plugin! i haven't knew it before. how have you see that mechanize is missing the accept-header and that the servers needs it ? was it only a guessing because firefox sends it? regards KoS -- Martin Kos +41-76-384-93-33 http://kos.liSay NO to HTML in mail ICQ# 13556143 Proudly running Debian GNU/Linux
Re: Mechanize - redirect problem
Hi Martin, I have written scrapers for a number of different sites, all of which require sign on, and have not had any problems with redirection not being performed. Hmm. Haven't done anything with sunrise yet (although I would like to automatically download my faxes from my sunrise onebox). One of the best tips I've gotten from this list is to put use LWP::Debug qw(+); in to your code. This turns on a trace so you can see what is happening. Have you looked at LWP::UserAgent::max_redirect()? By default (unless Mechanize changes this), it will only follow 7 redirects. By default it does not follow redirects in response to a post (which it is also not supposed to do, according to RFC). Hope this helps please do let me know how you fix it! Cheers, Peter Martin Kos wrote: hi as i've seen that this list is used for mechanize questions too i'll try my question... i try to login to the page http://mymobile.sunrise.ch/ but it seems like mechanize is not doing the redirect that is on the start site... if i try with my browser or wget i get redirect to a page like http://mymobile.sunrise.ch/portal/res/guest;jsessionid=HCCISJ1USYYSVQFIGZAXRAQ?paf_dm=fullpaf_gear_id=11?successURL=/portal/res/member%3Bjsessionid%3DHCCISJ1USYYSVQFIGZAXRAQ i tried it with a simple get but it doesn't work and i don't see what the problem could be... any idea what i'm doing wrong? btw. im using mechanize 1.08 regards KoS
Re: Mechanize - redirect problem
hi peter I have written scrapers for a number of different sites, all of which require sign on, and have not had any problems with redirection not being performed. Hmm. Haven't done anything with sunrise yet (although I would like to automatically download my faxes from my sunrise onebox). i have made a script to sent SMS over their site (so i don't need to login myself and i can use my local address book instead of entering all addresses in their online address book) and it worked fine until a week ago, it stopped working and i don't see why. One of the best tips I've gotten from this list is to put use LWP::Debug qw(+); in to your code. This turns on a trace so you can see what is happening. hey thanks! that helped a lot... now i see that mechanize get the first page GET http://mymobile.sunrise.ch/ and then gets redirected to GET http://mymobile.sunrise.ch/portal/res/member and then it gets a cookie extract_cookies: Set cookie JSESSIONID = ERXFXSMIMG5ZBQFIGZAXRA and it goes to the right URL GET http://mymobile.sunrise.ch/portal/res/guest;jsessionid=ERXFXSMIMG5ZBQFIGZAXRAQ?paf_dm=fullpaf_gear_id=11?successURL=/portal/res/member%3Bjsessionid%3DERXFXSMIMG5ZBQFIGZAXRAQ but instead of stopping at the url it gets an additional URL GET http://mobile.sunrise.ch/atg_500 and this site shows me an error site of sunrise. if i enter the long URL in my browser i see the normal login page that i should see! Have you looked at LWP::UserAgent::max_redirect()? By default (unless Mechanize changes this), it will only follow 7 redirects. By default it does not follow redirects in response to a post (which it is also not supposed to do, according to RFC). no it's definetly not the max_redirect, as it does to MUCH redirects ;-) Hope this helps please do let me know how you fix it! could you try a simple get on http://mymobile.sunrise.ch/ and see if you get the login-page instead of http://mobile.sunrise.ch/atg_500. perhaps i have a problem with my mechanize version (debian unstable) greets KoS btw.: i've just signed up for a /ch/open membership ;-) -- Martin Kos +41-76-384-93-33 http://kos.liSay NO to HTML in mail ICQ# 13556143 Proudly running Debian GNU/Linux
Re: Mechanize - redirect problem
On Wed, Feb 23, 2005 at 06:51:09PM +0100, Martin Kos ([EMAIL PROTECTED]) wrote: One of the best tips I've gotten from this list is to put use LWP::Debug qw(+); in to your code. This turns on a trace so you can Can one of you guys please write up a paragraph on that LWP::Debug trick so that I can drop it in the FAQ? I didn't even know about it. Thanks, xoxo, Andy -- Andy Lester = [EMAIL PROTECTED] = www.petdance.com = AIM:petdance
Mechanize - redirect problem
hi as i've seen that this list is used for mechanize questions too i'll try my question... i try to login to the page http://mymobile.sunrise.ch/ but it seems like mechanize is not doing the redirect that is on the start site... if i try with my browser or wget i get redirect to a page like http://mymobile.sunrise.ch/portal/res/guest;jsessionid=HCCISJ1USYYSVQFIGZAXRAQ?paf_dm=fullpaf_gear_id=11?successURL=/portal/res/member%3Bjsessionid%3DHCCISJ1USYYSVQFIGZAXRAQ i tried it with a simple get but it doesn't work and i don't see what the problem could be... any idea what i'm doing wrong? btw. im using mechanize 1.08 regards KoS -- Martin Kos +41-76-384-93-33 http://kos.liSay NO to HTML in mail ICQ# 13556143 Proudly running Debian GNU/Linux
