Yes, when I fail to get the page (simple is->success() test) I try again 5 times, and then give up and go to sleep again.
----- Original Message ----- From: "Gisle Aas" <[EMAIL PROTECTED]> To: "Beyond Control Inc." <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]>; "William R Ward" <[EMAIL PROTECTED]> Sent: Thursday, November 29, 2001 6:50 PM Subject: Re: 500 Read error > "Beyond Control Inc." <[EMAIL PROTECTED]> writes: > > > Unfortunatly, if I start it now it *will* work for 10 hours, then die. > > How many pages has been fetched at this point? 6 each hour for 12 > hours are 72, right? > > Does it fail quickly if you don't wait for 10 minutes between each > fetch? > > Regards, > Gisle > > > This is why I find this to be so weird, since I look like IE5, and I always > > create a new UserAgent object so that I won't get cookied or anything like > > that... > > > > ----- Original Message ----- > > From: "William R Ward" <[EMAIL PROTECTED]> > > To: <[EMAIL PROTECTED]> > > Sent: Wednesday, November 28, 2001 11:55 PM > > Subject: Re: 500 Read error > > > > > > > [EMAIL PROTECTED] (Beyond Control Inc.) writes: > > > > I've written a small script that fetches a page every 10 minutes. > > > > > > > > After 12-14 hours, it suddenly stops fetching the page, say there's a > > > > read error (500) > > > > > > Is this something that works once you start it up again? Or did it > > > work for a while and now you can't get it to work anymore? If the > > > latter, it could be that the web administrators noticed your script > > > and blocked it. Some web site owners don't like people slurping their > > > data when the ads or copyright notices might not be slurped as well. > >
