Re: [Python-Dev] Add sendfile() to core?
On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? So, was this proposal rejected? If so, for what reasons? Wrapper of such a useful call would be of great convenience, especially considering its availability on Win32 (see Thomas Heller's notice, [1]). Anyway, when one needs to send (arbitrarily large) files to a socket and back, ugly and slow workarounds are born, like this one: def copy_file(file1, file2, length, blocksize=40960): Transfer exactly length bytes from one file-like object to another sofar = 0 while sofar length: amount = blocksize if sofar + blocksize = length \ else length - sofar file2.write(file1.read(amount)) sofar += amount Using hypothetical os.sendfile() would be so much better! The only difficulty I can see is the choice of name for the wrapper. IMO, using sendfile from Linux and FreeBSD is pretty much okay; but objections may arise. [1] http://mail.python.org/pipermail/python-dev/2002-March/021543.html -- Sincerely, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On Sat, 08 Jan 2011 09:55:19 +0200 max ulidtko ulid...@gmail.com wrote: On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? So, was this proposal rejected? If so, for what reasons? I saw no patch for it, so Andrew probably didn't get to it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
Isn't that just shutil.copyfileobj()? On Fri, Jan 7, 2011 at 11:55 PM, max ulidtko ulid...@gmail.com wrote: On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? So, was this proposal rejected? If so, for what reasons? Wrapper of such a useful call would be of great convenience, especially considering its availability on Win32 (see Thomas Heller's notice, [1]). Anyway, when one needs to send (arbitrarily large) files to a socket and back, ugly and slow workarounds are born, like this one: def copy_file(file1, file2, length, blocksize=40960): Transfer exactly length bytes from one file-like object to another sofar = 0 while sofar length: amount = blocksize if sofar + blocksize = length \ else length - sofar file2.write(file1.read(amount)) sofar += amount Using hypothetical os.sendfile() would be so much better! The only difficulty I can see is the choice of name for the wrapper. IMO, using sendfile from Linux and FreeBSD is pretty much okay; but objections may arise. [1] http://mail.python.org/pipermail/python-dev/2002-March/021543.html -- Sincerely, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
A strong +1. Projects such as Twisted would certainly benefit from such an addiction. I'm not sure the os module is the right place for sendfile() to land though. Implementation between different platforms tends to vary quite a bit. A good resource is the samba source code which contains an implementation for all major UNIX systems. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/1/8 max ulidtko ulid...@gmail.com: On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? So, was this proposal rejected? If so, for what reasons? Wrapper of such a useful call would be of great convenience, especially considering its availability on Win32 (see Thomas Heller's notice, [1]). Anyway, when one needs to send (arbitrarily large) files to a socket and back, ugly and slow workarounds are born, like this one: def copy_file(file1, file2, length, blocksize=40960): Transfer exactly length bytes from one file-like object to another sofar = 0 while sofar length: amount = blocksize if sofar + blocksize = length \ else length - sofar file2.write(file1.read(amount)) sofar += amount Using hypothetical os.sendfile() would be so much better! The only difficulty I can see is the choice of name for the wrapper. IMO, using sendfile from Linux and FreeBSD is pretty much okay; but objections may arise. [1] http://mail.python.org/pipermail/python-dev/2002-March/021543.html -- Sincerely, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On Sun, 9 Jan 2011 11:17:40 -0800 Guido van Rossum gu...@python.org wrote: Isn't that just shutil.copyfileobj()? copyfileobj() still uses an user-space buffer (the Python bytes object used in the loop). The advantage of sendfile() is to bypass user-space logic and do the transfer entirely in kernel. How much it allows to gain *in practice* on a modern capable OS such as Linux I don't know. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
Am 09.01.2011 21:31, schrieb Antoine Pitrou: On Sun, 9 Jan 2011 11:17:40 -0800 Guido van Rossum gu...@python.org wrote: Isn't that just shutil.copyfileobj()? copyfileobj() still uses an user-space buffer (the Python bytes object used in the loop). The advantage of sendfile() is to bypass user-space logic and do the transfer entirely in kernel. How much it allows to gain *in practice* on a modern capable OS such as Linux I don't know. There would be at least two layers of savings: a) no Python objects would be created, and no bytecode loop would run the copying b) the data are not even copied into userspace at all My guess is that the savings of doing a) are larger than the savings of doing b). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
If you're gonna wrap sendfile, it might be nice to also wrap the splice, tee, and vmsplice syscalls on linux, since they're a lot more flexible. Also note that sendfile on BSD has a completely different signature to sendfile on linux. The BSD one has the rather odd functionality of a built-in writev() before and after the sending of the file itself, with an extra struct argument to specify that, while on linux, if you want to write some other buffers, you're just expected to call writev yourself. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On Sun, 9 Jan 2011 11:17:40 -0800, Guido van Rossum wrote: | Isn't that just shutil.copyfileobj()? | This function has two drawbacks. First, it copies until EOF, and has no possibility to copy exactly N bytes from source fd (say, opened socket). This is the reason why I (re)wrote my custom copying function - to allow that. The second drawback is not using huge performance bonus achievable with sendfile(2) syscall on some unices and TransmitFile() on win32. It allows for sending and receiving files at ethernet speeds with no CPU load at all, because no memory copying occurs (and no Python wrapper objects would be created for the buffers, no heap allocation, etc). All needed I/O requests are handled inside the kernel without a single copying of data. Thus for e.g. FTP servers using this syscall became a standard. It would be good if Python supported it. P.S. There seems to be a package which enables the sendfile support for Linux, FreeBSD and AIX, http://pypi.python.org/pypi/py-sendfile/. Though it's available only for 2.x. Hopefully I'll end up with a patch soon. --- Regards, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On 9 Jan, 08:09 pm, g.rod...@gmail.com wrote: A strong +1. Projects such as Twisted would certainly benefit from such an addiction. Eh. There would probably be some benefits, but I don't think they would be very large in the majority of cases. Also, since adding it to 2.x would be prohibited, it will be at least several years before Twisted actually benefits from its addition to the standard library. Plus, Pavel Pergamenshchik wrapped sendfile for Twisted already many years ago, but no one was interested enough to actually land the change in trunk. However, if it would help, I'm sure Pavel's code can be contributed to CPython. If anyone would like to take a look: http://twistedmatrix.com/trac/browser/branches/sendfile-585-4/twisted/python/test/test_sendfile.py http://twistedmatrix.com/trac/browser/branches/sendfile-585-4/twisted/python/_sendfile.c Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On 1/8/2011 2:55 AM, max ulidtko wrote: On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? There is no issue on the tracker and he apparently never did. There is a general policy of lightly wrapping useful os calls in os. Martin said this already in what was essentially a 'go ahead'. Problems include os differences, The only difficulty I can see is the choice of name for the wrapper. IMO, using sendfile from Linux and FreeBSD is pretty much okay; but objections may arise. [1] http://mail.python.org/pipermail/python-dev/2002-March/021543.html such as name differences (but I think *nix generally wins ;-), and the need for 'someone' to write the patches for the appropriate C-coded os files: posix, nt, os2, ce. Patch write makes initial decision on ironing out differences. The above was the second and last substantive answer to Andrew, as there was nothing much more to say. The tracker awaits ;-). Specify, if you can, whether you think the windows TransmitFile or modern equivalent is sufficiently compatible with the *nix sendfile to be wrapped with the same API or whether you propose Availability: Unix only. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
