This question is not related to Wget - so this newsgroup is probably not the
right place to post this message. However, I am posting this msg here hoping
that someone here might be able to help me.
I have just written a bare-bones crawler in C++. It seems to run just fine
except when getting data from some URLs. I noticed that the crawler took
about 30 seconds to return these pages to me (all other retrievals take less
than a second). The only thing common between these (30-sec) URLs is that
their response header-field contains "Etag". I would be grateful if someone
could help me with this problem : why do server responses with Etags take
longer ? Am I missing out on something in the request ?
This is the request I am sending out :
GET / HTTP/1.0
User-Agent:Tester
Host: srds.cs.umn.edu
Connection: Keep-Alive
I have also tried to retrieve those URLs using Wget and they don't take long
...
thanks
