Tan Joo Geok wrote:
>
> Hello.
>
> I have some problem downloading http://www.msn.com.sg.
> For both using lwp-request script from the libwww distribution and
> telnet directly to www.msn.com.sg on port 80, I get a error response
> of code 500, Internal Server Error. From IE and Netscape browsers,
> there is no problem with accessing this site though.
>
> Does anybody know why this is happening?
>
> Thank you for your attention.
>
> - Joo Geok
While watching the packets go by...
So the browser does a GET.
GET / HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
We get back a 302
Look at the 'location' header.
They also set a cookie.
HTTP/1.1 302 Object moved
Server: Microsoft-IIS/5.0
Date: Wed, 13 Dec 2000 16:49:36 GMT
Connection: keep-alive
Location: http://msid.msn.com/mps_id_sharing/redirect.asp?www.msn.com.sg/Default
Connection: Keep-Alive
Content-Length: 195
Content-Type: text/html
Set-Cookie: MC1=V=2&ID=8565B2350E6A46CD9E5C9518772E5667; expires=Sat, 04-Oct-200
Cache-control: private
Another GET for this 'new' page.
GET /mps_id_sharing/redirect.asp?www.msn.com.sg/Default.asp HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: msid.msn.com
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
and we receive another redirect,
as well as another cookie
HTTP/1.1 302 Object moved
Server: Microsoft-IIS/5.0
Date: Wed, 13 Dec 2000 16:49:37 GMT
Connection: keep-alive
Location: http://www.msn.com.sg/Default.asp?newguid=A06B0C8961384D3EB69D0CF64060
Connection: Keep-Alive
Content-Length: 195
Content-Type: text/html
Set-Cookie: MC1=V=2&GUID=A06B0C8961384D3EB69D0CF64060F42D; expires=Sat, 04-Oct-2
Cache-control: private
So we go get this page,
GET /Default.asp?newguid=A06B0C8961384D3EB69D0CF64060F42D HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&ID=8565B2350E6A46CD9E5C9518772E5667
we get another redirect and cookie.
HTTP/1.1 302 Object moved
Server: Microsoft-IIS/5.0
Date: Wed, 13 Dec 2000 16:49:38 GMT
Connection: keep-alive
Location: http://www.msn.com.sg/Default.asp
Connection: Keep-Alive
Content-Length: 154
Content-Type: text/html
Set-Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d; expires=Sat, 04-Oct-2
Cache-control: private
So now get that page.
GET /Default.asp HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
and another redirect
HTTP/1.1 301 Moved
Server: Microsoft-IIS/5.0
Date: Wed, 13 Dec 2000 16:49:37 GMT
Connection: keep-alive
Location: /homepage.asp
Connection: Keep-Alive
Content-Length: 0
Content-Type: text/html
Cache-control: private
So go get that page...
GET /homepage.asp HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
and finaly a 200.
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 13 Dec 2000 16:49:38 GMT
Connection: keep-alive
Connection: Keep-Alive
Content-Length: 72277
Content-Type: text/html
Expires: Wed, 13 Dec 2000 15:49:39 GMT
Cache-control: private
but wait,there's more...
GET /global/NS4.css HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
at least a 200.
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Connection: keep-alive
Date: Wed, 13 Dec 2000 16:49:38 GMT
Content-Type: text/css
Accept-Ranges: bytes
Last-Modified: Tue, 18 Apr 2000 12:45:25 GMT
ETag: "22b23f833a9bf1:86f"
Content-Length: 10975
Now the images...
GET /images/logo.gif HTTP/1.0
Referer: http://www.msn.com.sg/homepage.asp
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
and another one...
/GET /images/s.gif HTTP/1.0
Referer: http://www.msn.com.sg/homepage.asp
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
receiveing the image.
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Connection: keep-alive
Date: Wed, 13 Dec 2000 16:49:38 GMT
Content-Type: image/gif
Accept-Ranges: bytes
Last-Modified: Wed, 19 Jul 2000 15:40:04 GMT
ETag: "0a2729b97f1bf1:86f"
Content-Length: 2922
getting another one
GET /images/s.gif HTTP/1.0
Referer: http://www.msn.com.sg/homepage.asp
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
got one
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Connection: keep-alive
Date: Wed, 13 Dec 2000 16:49:38 GMT
Content-Type: image/gif
Accept-Ranges: bytes
Last-Modified: Wed, 08 Mar 2000 18:13:02 GMT
ETag: "b2f51ff12989bf1:86f"
Content-Length: 42
Get another one.
GET /images/space.gif HTTP/1.0
Referer: http://www.msn.com.sg/homepage.asp
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: www.msn.com.sg
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=a06b0c8961384d3eb69d0cf64060f42d
more images.
GET /ads/SGMPRE/005032830002_LG.gif HTTP/1.0
Referer: http://www.msn.com.sg/homepage.asp
Connection: Keep-Alive
User-Agent: Mozilla/4.72 [en] (X11; U; Linux 2.2.17 i686)
Host: ads.msn.com
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png
Accept-Encoding: gzip
Accept-Language: en
Accept-Charset: iso-8859-1,*,utf-8
Cookie: MC1=V=2&GUID=A06B0C8961384D3EB69D0CF64060F42D
and it goes on from here.
Your program has to do the same thing.
-Tim
