On May 6, 2010, at 11:26 AM, Igor wrote:
> Also, I think it's important, but is > said only here http://matforge.org/fipy/wiki/SnowLeopardSourceBuild > mpi4py must be present. Yep, we realized the same omission. Thanks for pointing it out. > After having all this I ran "mpirun -np 2 python setup.py test" and in > fact the execution time increased by few percent, and not decreased as > I expected, however, System Monitor showed the full load of 2 cores, > as opposed to normal run when only 1 core is seen to be used. The FiPy test suite is not a good benchmark for performance. It consists of lots of tiny problems that have lots of overhead. > So > behaved my own scripts - same (or slightly longer) execution time. How big is your mesh? > The > results of my scripts are same with or without mpirun, so it seems > that mpirun runs 2 instances of the script on 2 cores in parallel, and > not 1 instance with parallel execution (solution of the equation > system). No errors or anything in the output. The easiest way to check this is to run, e.g., mpirun -np 2 examples/diffusion/mesh1D.py You should get two viewers, regardless. If things are working properly, each viewer will display half the problem. If you're just getting two separate instances, as you suspect, then the two viewers should display the same thing. > > The questions I have: > 1) What could be wrong? As you surmised, I think your Trilinos is not compiled with MPI. Either that, or you are picking up an older version of PyTrilinos. Is there anything in your PYTHONPATH, LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH on Macs) that points to Trilinos 9, for instance? > 2) How can I check which solvers (pysparse/pytrilinos) are used by > fipy during the run? Here's one way: guyer$ python -c "from fipy import *; print DefaultAsymmetricSolver" --PySparse fipy.solvers.pysparse.linearLUSolver.LinearLUSolver guyer$ python -c "from fipy import *; print DefaultAsymmetricSolver" --Trilinos fipy.solvers.trilinos.linearGMRESSolver.LinearGMRESSolver > (Providing script with parameter --Trilinos or even some senseless to > fipy any word did not change the result and did not throw an error. By > the way, same was with FIPY_SOLVERS constant when I gave it different > values within a script) FiPy only looks for flags it recognizes, but it ignores flags it doesn't know (this is because there is no centralized place that parses all flags; components that care look for flags they know). If you are passing --Trilinos or setting FIPY_SOLVERS to trilinos, you should certainly be getting Trilinos. Also, if FiPy detects that it's running under multiple processors, it automatically switches to Trilinos. I think that should be true even if Trilinos is not MPI-enabled. > 3) Do you have any idea how to check if Trilinos was compiled with MPI > support? (it was installed by remote administrator) Try nm ${SITE_PACKAGES}/PyTrilinos/libpytrilinos.so | grep MPI (or libpytrilinos.dylib on a Mac). On serial builds, this returns nothing for me and on MPI builds, it returns a bunch of stuff. You'll need to figure out what ${SITE_PACKAGES} should be on your machine. > 4) If everything worked correct, would I really feel decrease of > execution time of my scripts by factor of ~2 in my case? We find the scaling to be generally linear, but that doesn't mean the constant of proportionality is 1. For small problems, there's so much MPI overhead that there's little or no benefit. With moderate to large problems, the boost is significant. These are rigorously statistical results, but if I run examples/phase/anisotropy.py on a 100x100 problem on my little 2-core laptop, the run time with 2 processors is about 77% what it is with 1 processor. For a 200x200 problem, the time with 2 processors is 71% of the 1-processor time.
