Re: [Python-Dev] Add sendfile() to core?
On Wed, 20 Mar 2002 14:53:58 -0500, Andrew Kuchling wrote: | sendfile() is used when writing really high-performance Web servers, | in order to save an unnecessary memory-to-memory copy. Question: | should I make up a patch to add a sendfile() wrapper to Python? So, was this proposal rejected? If so, for what reasons? Wrapper of such a useful call would be of great convenience, especially considering its availability on Win32 (see Thomas Heller's notice, [1]). Anyway, when one needs to send (arbitrarily large) files to a socket and back, ugly and slow workarounds are born, like this one: def copy_file(file1, file2, length, blocksize=40960): Transfer exactly length bytes from one file-like object to another sofar = 0 while sofar length: amount = blocksize if sofar + blocksize = length \ else length - sofar file2.write(file1.read(amount)) sofar += amount Using hypothetical os.sendfile() would be so much better! The only difficulty I can see is the choice of name for the wrapper. IMO, using sendfile from Linux and FreeBSD is pretty much okay; but objections may arise. [1] http://mail.python.org/pipermail/python-dev/2002-March/021543.html -- Sincerely, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add sendfile() to core?
On Sun, 9 Jan 2011 11:17:40 -0800, Guido van Rossum wrote: | Isn't that just shutil.copyfileobj()? | This function has two drawbacks. First, it copies until EOF, and has no possibility to copy exactly N bytes from source fd (say, opened socket). This is the reason why I (re)wrote my custom copying function - to allow that. The second drawback is not using huge performance bonus achievable with sendfile(2) syscall on some unices and TransmitFile() on win32. It allows for sending and receiving files at ethernet speeds with no CPU load at all, because no memory copying occurs (and no Python wrapper objects would be created for the buffers, no heap allocation, etc). All needed I/O requests are handled inside the kernel without a single copying of data. Thus for e.g. FTP servers using this syscall became a standard. It would be good if Python supported it. P.S. There seems to be a package which enables the sendfile support for Linux, FreeBSD and AIX, http://pypi.python.org/pypi/py-sendfile/. Though it's available only for 2.x. Hopefully I'll end up with a patch soon. --- Regards, max ulidtko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
