On May 7, 2010, at 5:37 PM, jtg wrote:
> First of all, many thanks to Dr. Guyer for the very clear diagnostics > for checking on the status of FiPy/Trilinos. I suspect that the > effort required of the development team to get Trilinos to work > with some transparency from within FiPy was likely substantial. > The simple steps he outlined, helping us users to figure out what's > wrong on our side, are most welcome. You're welcome. While we're pleased with the transparency of *running* FiPy in parallel, we're not at all satisfied with the transparency of getting it installed in the first place. > I am unsure, however, about the implications of this reference to > "Trilinos 9" -- is 9 Good or Bad? (Probably should be obvious to > me, but isn't. I'll guess "Bad".) Following Dr. Guyer's checkpoints > in his e-mail to Igor... Sorry. That was sloppily written on my part. What I meant is that if you're not careful with environment variables (specifically, PYTHONPATH and (DY)LD_LIBRARY_PATH), you can end up loading an older version of PyTrilinos. Since FiPy did not previously take advantage of multiple processors, it seemed plausible that many FiPy users would have an older PyTrilinos that was not compiled with MPI support. Trilinos 9 is supported and is what we used when writing the parallel support in FiPy 2.1. Trilinos 10.2 has also been successfully tested. > o running nm against "libpytrilinos.so" produces *no* references to > "MPI" (sigh, also Not Good). however, I am puzzled because all > the other trilinos libs > (e.g., .../trilinos/lib/python2.6/site-packages/PyTrilinos/_Teuchos.so) > *do* have MPI references Yeah, the `nm` test I suggested is adequate in this instance, I think, but it's not really proper. In this case, I think it happens that _Teuchos.so has some routines that refer to MPI, whether or not MPI is linked in. __ZN7Teuchos16GlobalMPISession8getNProcEv, for instance, probably returns 1 if MPI isn't present and returns whatever is proper if it is present. We do something similar in FiPy's Python code. Instead, to see if OpenMPI library is linked in, on a Mac, try otool -L .../site-packages/PyTrilinos/libpytrilinos.dylib | grep libmpi On Debian, this works ldd .../site-packages/PyTrilinos/libpytrilinos.so | grep libmpi If you compiled against a different MPI library, then you'll need to grep for that. > o but I also have Trilinos 9.0.3 -- so I'm going to guess that this is > also Not Good? That should be fine. > Compiling and > installing these beasts (especially Trilinos) is sufficiently fraught > with peril and error that I would be very happy if l could avoid doing > so. But if I must... I completely understand your reticence. I do find Trilinos 10.2 to be less painful to build than 9.0 (after much initial struggle with cmake). > So, before I start down the re-installation path, I would feel better if > I could see a list of releases for these required components that are > compatible with parallel procesing support for FiPy 2.1. If someone > could please direct me to or provide such a list, I would be very grateful. I'm afraid an exhaustive list of all acceptable configurations is beyond our resources. I can tell you what I'm presently running though: Mac OS X 10.6.3 "Snow Leopard" OpenMPI 1.4.1 Trilinos 10.2.0 mpi4py 1.2.1 We did most of the parallel development on a Mac OS X 10.5.x "Leopard" machine with Trilinos 9.0.x. I no longer have that machine, so cannot easily tell you what versions of other packages were present. I have tried (not very hard) but not yet succeeded in building Trilinos 9 for Snow Leopard. Wheeler can post with whatever versions of things he's gotten to work with Debian and/or Ubuntu. Ultimately, I don't think things are too sensitive to version #'s, but Trilinos is quite sensitive to configuration flags. In this case, if your libpytrilinos.so doesn't have any "MPI" symbols in it (and certainly if ldd shows that an MPI library isn't linked in), then that's why you're getting two independent processes and you're going to have to rebuild PyTrilinos.
