Re: [Python-Dev] Darwin's realloc(...) implementation never shrinks allocations
On Thu, Jan 01, 1970 at 12:00:00AM +, Tim Peters wrote: > Is there any known case where Python performs poorly on this OS, for > this reason, other than the "pass giant numbers to recv() and then > shrink the string because we didn't get anywhere near that many bytes" > case? > > [...] > > I agree the socket-abuse case should be fiddled, and for more reasons > than just Darwin's realloc() quirks. [...] Yes, in the socket-abuse > case, where the program routinely malloc()s strings millions of bytes > larger than the socket can deliver, it would obviously help. That's > not typically program behavior (however typical it may be of that > specific app). Note that, with respect to http://python.org/sf/1092502, the author of the (original) program was using the documented interface to a file object. It's _fileobject.read() that decides to ask for huge numbers of bytes from recv() (specifically, in the max(self._rbufsize, left) condition). Patched to use a fixed recv_size, you of course sidestep the realloc() nastiness in this particular case. bacchusrx. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Darwin's realloc(...) implementation never shrinks allocations
On Mon, Jan 03, 2005 at 03:55:19PM -0500, Bob Ippolito wrote: > >Note that, with respect to http://python.org/sf/1092502, the author > >of the (original) program was using the documented interface to a > >file object. It's _fileobject.read() that decides to ask for huge > >numbers of bytes from recv() (specifically, in the > >max(self._rbufsize, left) condition). Patched to use a fixed > >recv_size, you of course sidestep the realloc() nastiness in this > >particular case. > > While using a reasonably sized recv_size is a good idea, using a > smaller request size simply means that it's less likely that the > strings will be significantly resized. It is still highly likely they > *will* be resized and that doesn't solve the problem that > over-allocated strings will persist until the entire request is > fulfilled. You're right. I should have said, "you're more likely to get away with it." The underlying issue still exists. My point is that the problem is not analogous to the guy who tried to read 2GB directly from a socket (as in http://python.org/sf/756104). Googling for MemoryError exceptions, you can find a number of spurious problems on Darwin that are probably due to this bug: SpamBayes for instance, or the thread at http://mail.python.org/pipermail/python-list/2004-November/250625.html bacchusrx. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
