Hi,
On Tue, Feb 17, 2009 at 07:20:04PM -0500, James Antill wrote:
> On Sun, 2009-02-15 at 23:57 -0500, Andrew Trusty wrote:
> > So I've been using urlgrabber for awhile in a number of projects and I
> > love it because it usually works great. In my most recent project I'm
> > using it to access Google's ajax translation api. Since I'm
> > translating a lot of stuff I keep urlgrabber a lot busier than in my
> > previous projects. I noticed though, that after fetching translations
> > for around 1000 words my python process would run out of available
> > file descriptors to use and urlgrabber and other parts of the code
> > that used files would fail. I figured out that urlgrabber was keeping
> > a socket open for each word I translated since keepalive is on by
> > default. So my current simple fix is to pass the close_connection=1
> > keyword argument to make keepalive close its sockets.
> >
> > But I'm curious why I have to do this. Isn't the point of keepalive
> > that further requests (all of which go to the google translation
> > servers) should use the same socket and not open new sockets? Or do I
> > need to use urlgrabber in a different way (currently I'm just using
> > the urlread with the default_grabber)?
>
> Search bugzilla.redhat.com ... there are "bugs" in python/urllib2 which
> leaks file descriptors.
I noticed this bug several times, but never really pursued it. I
basically run into it each time I do a full system update that involves
download of > ~1000 packages. (A simple re-run works around nicely...)
And I always wondered whether you guys, using yum much more frequently
than me, have a patch in python/urllib2 to work around this? I never
asked but repeatedly wondered... so now... have you? :-)
Peter
--
Contact: ad...@opensuse.org (a.k.a. ftpad...@suse.com)
#opensuse-mirrors on freenode.net
Info: http://en.opensuse.org/Mirror_Infrastructure
SUSE LINUX Products GmbH
Research & Development
_______________________________________________
Yum-devel mailing list
Yum-devel@lists.baseurl.org
http://lists.baseurl.org/mailman/listinfo/yum-devel
