All the discussion recently about pyprocessing got me interested in
actually benchmarking Python's multiprocessing performance to see if
reality matched my expectations around what would scale up and what
would not. I knew Python threads wouldn't be good for compute bound
problems, but I was curious to see how well they worked for i/o bound
problems. The short answer is that for i/o bound problems, python
threads worked just as well as using multiple operating system
processes.
I wrote two simple benchmarks, one compute bound and the other i/o
bound. The compute bound one did a parallel matrix multiply and the i/
o bound one read random records from a remote MySQL database. I ran
each benchmark via python's thread module and via MPI (using mpi4py
and openmpi and Send()/Recv() for communication). Each test was run
multiple times and the numbers were consistent between test runs. I
ran the tests on a dual-core Macbook Pro running OS X 10.5 and the
included python 2.5.1.
1) Python threads
a) compute bound:
1 thread - 16 seconds
2 threads - 21 seconds
b) i/o bound:
1 thread -- 13 seconds
4 threads -- 10 seconds
8 threads -- 5 seconds
12 threads - 4 seconds
2) MPI
a) compute bound:
1 thread - 17 seconds
2 threads -- 11 seconds
b) i/o bound
1 thread -- 13 seconds
4 threads -- 10 seconds
8 threads -- 6 seconds
12 threads -- 4 seconds
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
