MRAB wrote:
dq wrote:
MRAB wrote:
dq wrote:
Martin v. Löwis wrote:
So does anyone know what the deal is with this? Why is the
same code so much slower on Windows? Hope someone can tell
me before a holy war erupts :-)
Only the holy war can give an answer here. It certainly has
*nothing* to do with Python; Python calls the operating system
functions to read from the network and write to the disk almost
directly. So it must be the operating system itself that slows
it down.
To investigate further, you might drop the write operating, and
measure only source.read(). If that is slower, then, for some
reason, the network speed is bad on Windows. Maybe you have the
network interfaces misconfigured? Maybe you are using wireless
on Windows, but cable on Linux? Maybe you have some network
filtering software running on Windows? Maybe it's just that
Windows sucks?-)
If the network read speed is fine, but writing slows down, I
ask the same questions. Perhaps you have some virus scanner
installed that filters all write operations? Maybe Windows
sucks?
Thanks for the ideas, Martin. I ran a couple of experiments to find
the culprit, by downloading the same 20 MB file from the
same fast server. I compared:
1. DL to HD vs USB iPod.
2. AV on-access protection on vs. off
3. "source. read()" only vs. "file.write( source.read() )"
The culprit is definitely the write speed on the iPod. That is,
everything runs plenty fast (~1 MB/s down) as long as I'm not
writing directly to the iPod. This is kind of odd, because if I
copy the file over from the HD to the iPod using windows
(drag-n-drop), it takes about a second or two, so about 10 MB/s.
So the problem is definitely partially Windows, but it also seems
that Python's file.write() function is not without blame. It's
the combination of Windows, iPod and Python's data stream that is
slowing me down.
I'm not really sure what I can do about this. I'll experiment a
little more and see if there's any way around this bottleneck.
If anyone has run into a problem like this, I'd love to hear
about it...
You could try copying the file to the iPod using the command line,
or copying data from disk to iPod in, say, C, anything but Python.
This would allow you to identify whether Python itself has anything
to do with it.
Well, I think I've partially identified the problem. target.write(
source.read() ) runs perfectly fast, copies 20 megs in about a
second, from HD to iPod. However, if I run the same code in a while
loop, using a certain block size, say target.write( source.read(4096)
), it takes forever (or at least I'm still timing it while I write
this post).
The mismatch seems to be between urllib2's block size and the write
speed of the iPod, I might try to tweak this a little in the code and
see if it has any effect.
Oh, there we go: 20 megs in 135.8 seconds. Yeah... I might want to
try to improve that...
How long does it take to transfer 4KB? If it can transfer 1MB/s then I'd
say that 4KB is too small. Generally speaking, the higher the data rate,
the larger the blocks you should be transferring at a time, IMHO.
You could write a script to test the transfer speed with different block
sizes.
Thanks MRAB, 32 or 64 KB seems to be quickest, but I'll do a more
scientific test soon and see what turns up.
--
http://mail.python.org/mailman/listinfo/python-list
