> > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 14 Dec 2011 08:44:37 +0000 > From: Leo Lapworth <[email protected]> > Subject: Re: Perl threads and libwww wierdness > To: "London.pm Perl M[ou]ngers" <[email protected]> > Message-ID: > <ca+rmup49p3y6v8vmzy03i26qwgotzmtvand+eopzg2fgjmp...@mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > I still think it's your webserver.... > > On 14 December 2011 01:16, Toby Wintermute <[email protected]> wrote: > > I'm hitting some really odd behaviour, infrequently, with libwww and > > mechanize under a highly-threaded Perl. > > > I have encountered a situation where I see unusual 404 errors - in > > between 0.03% to 0.10% of requests. > > Errors are randomly spaced on random pages > > > > The 404 errors are reported on the distant webserver as well, for URLs > > that are definitely not 404. (as the identical URL is being requested > > successfully many times in the same period). > > My logic would be, if the webserver is reporting intermittent 404's for a > specific URL, then it's the webserver that's generating it. >
In general, I'd agree with Leo, although there may be wrinkles in the HTTP request you aren't seeing (cache headers). Have you tried a detailed dump of each outbound request? If the service owner can't give you a reason for the 404s on a valid URL, then you might consider a proxy to get more detailed diagnostics on the request/response. Ash > > > Scale: This is typically running around 40 threads, all going flat-out > > on an 8-core system; issues show up whenever you get over ~6 threads > > though. > > So the webserver can't cope with the traffic generated when you use > more than 6 threads, so too many requests. > > > The only reason I don't think this is a problem with the network or > > webserver is that the problems don't show up if I use fork() instead > > of threads. (On otherwise identical code; and the same overall > > throughput rates are reached. However the fork() version is just for > > that bit of code for testing this; it misses some functionality.) > > Does the fork() submit the same number of requests in the same > time period? - if it's less than your thread version then that > would point to the webserver, not the Perl code. > > I'd look into if your webserver (you don't mention what software it > is, apache/starman/IIS?) has some sort of "arrrrgggg, I can't cope, > throw a 404" setting or bug. > > Good luck! > > Leo > >
