I don't think using a sniffer is the best option. A web paged is built
up from multiple objects and the browser has to request each object in
turn. Just count the pictures on the login screen, there are quite a few
pictures there!
Prep things up like this. On a form, put a TIdHTTP a Memo and a
TIdCookieManager. Make sure you add "IdMultiPartFormData" to the uses
clause.
Also add two buttons, call the first one "Login" and the second one
"Test".
I've named my TIdHTTP component "H" - not nice but easy typing.
In the "Test" button OnClick add this code:
<code>
Memo1.Lines.Text :=
H.Get('http://www.morningstar.com/hp.html?byrefresh=yes');
</code>
In the "Login" button OnClick add this code:
<code>
procedure TForm1.Button1Click(Sender: TObject);
var Data:TIdMultiPartFormDataStream;
begin
Memo1.Lines.Text :=
H.Get('http://members.morningstar.com/DotNet/LogIn.aspx?pgid=hetoplogin'
);
Memo1.Lines.SaveToFile('C:\Temp\1.htm');
Data := TIdMultiPartFormDataStream.Create;
try
Data.AddFormField('email', '[EMAIL PROTECTED]');
Data.AddFormField('password', 'yourpassword1');
Data.AddFormField('url', '/');
Memo2.Lines.Text :=
H.Post('http://members.morningstar.com/DotNet/LogIn.aspx', Data);
Memo2.Lines.SaveToFile('C:\Temp\2.htm');
finally Data.Free;
end;
end;
</code>
Start the program. Press the "Test" button. Copy the resulting code in
Memo1 to Notepad. Do a search for your name. If you'd be logged in your
name would show up in that code, because of the "Welcom Username" text
on the page.
Next press the "Login" button. You'll get an error. Ignor the error :-)
Now press the "Test" button and repeat the copy/paste/search bits again.
The "Welcome Username" text is on the page so that's proof you're in :-)
Now if you need to do it all over again, here's how you'd do it:
(1) You go to the main page of www.morningstar.com; Wait for all the
welcome stuff to go away, then click on the "Login" button at the top of
the page. The URL you'll see is the URL I used in the first "Get". You
need to do that first "Get" so all the cookies are properly set up.
(2) Take a look at the source for the page you're viewing. Do a search
for the login form. It's easy, just search for "<FORM". You'll notice
there are three forms on the page, you'll need to guess the correct one.
(3) Note the "ACTION" URL in the FORM tag. That's the URL I used in the
"POST".
(4) Remove everything from the HTML document that's in front of the FORM
you're dealing with. This is obviously not required but it makes
searching a bit easyer.
(5) Do a search for "</FORM" - that markes the end of the HTML form. Now
delete everything after that tag, so you'd be left with only the login
form on screen.
(6) Now do a serch for "<INPUT" - this should reveal the names for every
single field in the form. There's a lot of extra in there, mostly table
tags, just ignore everything. You'll get the names of the login fields
(email and password), just as I found them. You'll also note there's an
hidden field (the "url") field - you'll have to pass that back to the
login handler (the post url) just in case it checks for that.
(7) You're all set up, you may now login from your Delphi application
using your TIdHTTP component.
P.S: I *think* the error you're getting when you login is related to the
way the ASPX on the server side handles the succesfull login. I think it
sends a redirect that TIdHTTP can't handle but it doesn't seem to
matter, since you are actually logged in and you may start reading
pages.
P.S.2: I tried setting TIdHTTP.HandleRedirects=True - it doesn't remove
the error, it only changes the error number.
P.S.3: You may get rid of the error by surrounding your login code with
an try-except block.
--
Cosmin Prund
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
> Behalf Of Rainer von Saleski
> Sent: 15 iulie 2007 23:43
> To: [email protected]
> Subject: Re: Cookies
>
> Hi Cosmin,
>
> What you have suggested is basically what I have tried -- but it does
> not seem to work for www,morningstar.com. I have used an HTTP sniffer
> (by Effetech) to capture the traffic involved in a login process (over
> 100 transfers including graphics and ads!), and then tried sending the
> pertinent "Get" requests in a new session. I have tried this from
> Delphi, and by pasting the strings into FireFox's Address bar. I
> simply
> cannot get it to work; there seem to be some cookie interactions that
> I'm not accounting for, even though I'm repeating everything sent
> (knowingly) from my end in a successful "manual" login.
>
> So ... what I need ... is a browser Delphi component that uses TIdHTTP
> (and hence TIdCookieManager), so I can let the user carry out the
> login,
> and then conduct the rest of my information retrieval programatically.
>
> Any suggestions, anybody?
>
> Rainer
>
>
> Cosmin Prund wrote:
> >
> > I think you sent this message to my private address in error.
> >
> > I forwarded the message to the list so everyone can see it.
> >
> >
> >
> > I'm suggesting automating the login using TiDHTTP. Unless the login
> > screen requires JavaScript (sometimes even if it does require
> > JavaScript), you can automate the process. Here's an example of what
> > I'm thinking of:
> >
> >
> >
> > procedure LogIn;
> >
> > var L:TStringList;
> >
> > begin
> >
> > HTTP.Get('login-screen.php');
> >
> > L := TStringList.Create;
> >
> > Try
> >
> > L.Add('username=myusername');
> >
> > L.Add('password=mypassword');
> >
> > HTTP.Post('login-screen-processor.php', L);
> >
> > Finally L.Free;
> >
> > End;
> >
> > end;
> >
> >
> >
> > You essentially start at the login-screen. This is the URL you'd
type
> > in a browser to get to the site. Next you'd setup a "response" and
> > post it back to the server. Depending on how security is
implemented,
> > you'll need to parse the result you got for the "Get" statement and
> > add to your "response" strings all of the fields in the login form.
> > You might also need to parse the result to find the actual "action"
> > for the login form. None of this should be that difficult...
> >
> >
> >
> > Can you post the URL to the site you want to automatically log on to
> > so we can take a look at it, and maybe suggest a different route?
> >
> >
> >
> > --
> >
> > Cosmin Prund
> >
> >
> >
> > *From:* Rainer von Saleski [mailto:[EMAIL PROTECTED]
> > *Sent:* Thursday, July 05, 2007 11:23 PM
> > *To:* Cosmin Prund
> > *Subject:* Re: Delphi Digest, Vol 54, Issue 3
> >
> >
> >
> > Thanks for the reply, Cosmin!
> >
> > Unfortunately, the TWebBrowser component seems to have nothing to do
> > with TIdHTTP ... so using TWebBrowser to do the login does not
create
> > cookies where TIdHTTP can find them, and I cannot login that way.
> >
> > Simply sending the login URL with TIdHTTP does not work either ...
> > apparently there is some preparation that goes on before getting to
> > the login screen that is needed.
> >
> > Does Indy have an equivalent (in part or in whole) of the
> > functionality of TWebBrowser that uses TIdHTTP rather than the
> Windows
> > DLL? I don't seem to be able to find it.
> >
> > Thanks in advance,
> > Rainer
> >
> >
> >
> >
> >
---------------------------------------------------------------------
> ---
> >
> >
> >
> > Subject:
> >
> > RE: Cookies
> >
> > From:
> >
> > "Cosmin Prund" <[EMAIL PROTECTED]>
> > <mailto:[EMAIL PROTECTED]>
> >
> > Date:
> >
> > Tue, 3 Jul 2007 10:21:04 +0300
> >
> > To:
> >
> > "Borland's Delphi Discussion List" <[email protected]>
> > <mailto:[email protected]>
> >
> >
> >
> > To:
> >
> > "Borland's Delphi Discussion List" <[email protected]>
> > <mailto:[email protected]>
> >
> >
> >
> > From: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
> [mailto:[EMAIL PROTECTED] On
> > Behalf Of Rainer von Saleski
> > Sent: Monday, July 02, 2007 6:12 PM
> > To: [email protected] <mailto:[email protected]>
> > Subject: Cookies
> >
> > I'm doing just fine using HTTP.Get (Delphi 7) to fetch the web data
I
> > need ... except that I cannot figure out how to set up the cookies
> >
> > that
> >
> >
> > the site I'm contacting needs to log me in.
> >
> >
> >
> > Indy stores its cookies in its own place ... but where is that?
> If I
> >
> > knew, I could just copy in Firefox's cookies for that site.
> >
> >
> >
> > If I use TWebBrowser to let the user carry out a login, the
> cookies do
> >
> > not end up in Indy's world ... and I'm not logged in for
> HTTP.Get.
> >
> >
> >
> > The site's login page is sufficiently complicated, I cannot
> figure out
> >
> > what URL to send to simply do the login. Is there a way to
> "sniff"
> >
> >
> >
> > the
> >
> >
> > traffic between a browser (any browser) and the internet?
> >
> >
> >
> > Answers to any of these avenues will probably solve my problem!
> >
> >
> >
> > Thanks,
> >
> > Rainer
> >
> >
> >
> >
> > Hello there!
> >
> > Indy has its own cookie manager and you should use that:
> > TIdCookieManager (it's on the Indy Misc page); Once you hook it up
to
> > the HTTP component you've got nothing else to do, it just works. As
> for
> > copying the cookie from Firefox... it might work, it might not work.
> It
> > depends on how the cookie is constructed and how it is validated.
> >
> > It just might be easier to do the login from Delphi + Indy (this
> might
> > also be more portable) - but you'd need a bit of HTML knowledge to
do
> > this! Not much knowledge (I did it once and I can't say I know HTML
> or
> > JavaScript).
> >
> > --
> > Cosmin Prund
> >
> >
> >
> >
---------------------------------------------------------------------
> ---
> >
> >
> >
> >
> > _______________________________________________
> > Delphi mailing list -> [email protected] <mailto:[email protected]>
> > http://www.elists.org/mailman/listinfo/delphi
> >
> _______________________________________________
> Delphi mailing list -> [email protected]
> http://www.elists.org/mailman/listinfo/delphi
_______________________________________________
Delphi mailing list -> [email protected]
http://www.elists.org/mailman/listinfo/delphi
