On Tue, Sep 9, 2008 at 12:23 PM, Micah Cowan <[EMAIL PROTECTED]> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Donald Allen wrote: > > On Tue, Sep 9, 2008 at 3:14 AM, Daniel Stenberg <[EMAIL PROTECTED]> wrote: > >> On Mon, 8 Sep 2008, Donald Allen wrote: > >> > >>> The page I get is what would be obtained if an un-logged-in user went > to > >>> the specified url. Opening that same url in Firefox *does* correctly > >>> indicate that it is logged in as me and reflects my customizations. > >> First, LiveHTTPHeaders is the Firefox plugin everyone who tries these > stunts > >> need. Then you read the capure and replay them as closely as possible > using > >> your tool. > >> > >> As you will find out, sites like this use all sorts of funny tricks to > >> figure out you and to make it hard to automate what you're trying to do. > >> They tend to use javascripts for redirects and for fiddling with cookies > >> just to make sure you have a javascript and cookie enabled browser. So > you > >> need to work hard(er) when trying this with non-browsers. > >> > >> It's certainly still possible, even without using the browser to get the > >> first cookie file. But it may take some effort. > > > > I have not been able to retrieve a page with wget as if I were logged > > in using --load-cookies and Micah's suggestion about 'Accept-Encoding' > > (there was a typo in his message -- it's 'Accept-Encoding', not > > 'Accept-Encodings'). I did install livehttpheaders and tried > > --no-cookies and --header <cookie info from livehttpheaders> and that > > did work. > > That's how I did it as well (except I got the headers from tcpdump); I'm > using Firefox 3, so don't have access to FF's new sqllite-based cookies > file (apart from the patch at > > http://wget.addictivecode.org/FrontPage?action=AttachFile&do=view&target=wget-firefox3-cookie.patch > ). > > > Some of the cookie info sent by Firefox was a mystery, > > because it's not in the cookie file. Perhaps that's the crucial > > difference -- I'm speculating that wget isn't sending quite the same > > thing as Firefox when --load-cookies is used, because Firefox is > > adding stuff that isn't in the cookie file. Just a guess. > > Probably there are session cookies involved, that are sent in the first > page, that you're not sending back with the form submit. > - --keep-session-cookies and --save-cookies=<foo.txt> make a good > combination. > > > Is there a > > way to ask wget to print the headers it sends (ala livehttpheaders)? > > I've looked through the options on the man page and didn't see > > anything, though I might have missed it. > > - --debug Well, I rebuilt my wget with the 'debug' use flag and ran it on the yahoo test page (after having logged in to yahoo with firefox, of course) with --load-cookies and the accept-encoding header item, with --debug. Very useful. wget is sending every cookie item in firefox's cookies.txt. But firefox sends three additional cookie items in the header that wget does not send. Those items are *not* in firefox's cookies.txt so wget has no way of knowing about them. Is it possible that firefox is not writing session cookies to the file? The result of this test, just to be clear, was a page that indicated yahoo thought I was not logged in. Those extra items firefox is sending appear to be the difference, because I included them (from the livehttpheaders output) when I tried sending the cookies manually with --header, I got the same page back with wget that indicated that yahoo knew I was logged in and formatted with page with my preferences. /Don > > > - -- > HTH, > Micah J. Cowan > Programmer, musician, typesetting enthusiast, gamer. > GNU Maintainer: wget, screen, teseq > http://micah.cowan.name/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.7 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iD8DBQFIxqL77M8hyUobTrERAovFAJ9yagS2xW+2wFG65BwiFkJNfTMylgCfYaq7 > 1vOmTDimFg8E7Cn+Q+HGZn8= > =JKXH > -----END PGP SIGNATURE----- >
