Looks like uedge build tools look for mpiuni spearately and adds in -Impiuni.
But I think its good to keep -Impiuni in petsc makefiles for regular users who have mpi.h [or mpif.h] in their non-petsc soures. Satish On Fri, 26 Feb 2010, Barry Smith wrote: > > Satish, > > Ok I will expose the -Impiuni/ so that external packages can use the MPI uni > directly. Please let me know if problems arise. > > Barry > > On Feb 26, 2010, at 2:01 PM, Satish Balay wrote: > > > I should have included facets-devel in this discussion. Will cc > > facets-devel list now [there are likely to be e-mail bounces due to > > this - will handle the bounces on petsc-dev] > > > > The discussion is at: > > http://lists.mcs.anl.gov/pipermail/petsc-dev/2010-February/002200.html > > http://lists.mcs.anl.gov/pipermail/petsc-dev/2010-February/002270.html > > > > The one usage I know off - is petsc+mpiuni is from uedge. I'm guessing > > there are other usages from facets [I don't really know these other > > usages] > > > > Based on the previous discussions on facets-devel list, I was > > advocating the following: when having sequential builds with multiple > > packages - that have their own internal seqmpi switch [with normal mpi > > code]: > > > > - the best thing to do is always use a proper mpi impl even for > > sequential builds - and make every package use this impl[ with > > perhaps an error message when parallel run is attempted] > > > > - if proper mpi must to be avoided - then one way to minise conflicts > > is use seq-mpi from only one package - and have others use this > > aswell. However this does not work with all packages [as there are > > some packages that do need MPI - like hypre etc..]. > > > > So currently mpiuni is being used as the common seq-mpi [from uedge, > > and perhaps other codes] > > > > Wrt uedge I'm yet to check how the current changes might affect > > it. [currently uedge buildsystem does not work with petsc-dev [due to > > the makefile changes]. will check on this later. > > > > On Fri, 26 Feb 2010, Barry Smith wrote: > > > > > > > > If we switch back slightly to the old model in the following way will that > > > resolve the problems with FACETS? > > > > > > 1) expose the mpiuni include directory by adding the mpiuni include path > > > to > > > the list of include search paths > > > > wrt uedge - the current buildsystem picks up everything from PETSc makefiles > > - > > so the above change should work. > > > > > 2) DO NOT set PETSC_HAVE_MPI > > > > uedge does not use this - so this change should be fine. > > > > > 3) DO NOT have a separate mpiuni library (which doesn't exist with single > > > library anyways). > > > > since uedge links with petsc+mpi detected from petsc makefiles - this > > should work.. > > > > > Slightly related note: is there any reason not to always #if > > > defined(MPIUNI_AVOID_MPI_NAMESPACE) so that C mpi symbols don't appear in > > > our > > > libraries. Not that this is important but I still like simple clean code > > > without a bunch of confusing if not needed ifdefs. > > > > Looks like when this flag is defined - the fortran mpiuni symbols are > > also not built. > > > > satish > > > > > > > > > > > > > Barry > > > > > > On Feb 26, 2010, at 1:00 PM, Barry Smith wrote: > > > > > > > > > > > 1) So FACETS Fortran code includes mpif.h directly AND includes PETSc > > > > includes in the same file? Can those extra includes just be removed from > > > > FACETS. > > > > 2) Some FACES codes, or other packages that FACETS use THAT DO NOT use > > > > PETSc can/do use PETSc's mpiuni mpif.h? > > > > > > > > Are these ONLY conflict with the new MPI unimodel and FACETS or are > > > > there > > > > other conflicts? > > > > > > > > Is there any issue with not having a separate libmpiuni.a ? > > > > > > > > Barry > > > > > > > > On Feb 26, 2010, at 12:46 PM, Satish Balay wrote: > > > > > > > > > On Fri, 26 Feb 2010, Barry Smith wrote: > > > > > > > > > > > > > > > > > On Feb 26, 2010, at 11:22 AM, Satish Balay wrote: > > > > > > > > > > > > > I think eliminating mpi.h, mpi.mod etc is not a good change. Its > > > > > > > likely to break users codes. I suspect it breaks FACETS. [and > > > > > > > perhaps > > > > > > > Flotran] > > > > > > > > > > > > Suspect isn't good enough. HOW does it break FACETS? It won't break > > > > > > pflotran > > > > > > unless they have #include "mpi.h" directly in their code > > > > > > > > > > Just for this reason [prohibiting using 'mpi.h'/mpif.h from user code > > > > > - if one wants --with-mpi=0] the current change is bad > > > > > > > > > > > > > > > I've explained the facets/uedge model used below. I'll check uedge > > > > > usage. I'll have to ask facets folks to check other usages.. > > > > > > > > > > > > > > > > > > > With this change we will continue to have MPI symbols [esp from > > > > > > > fortran] in libpetsc.a. So this is not really clean absorbtion of > > > > > > > mpiuni into petsc. And I don't think we can avoid having these MPI > > > > > > > symbols in -lpetsc. > > > > > > > > > > > > Yes the fake "symbols" are in libpetsc.a but what is wrong with > > > > > > that? > > > > > > What > > > > > > is the disadvantage over having them in libmpiuni.a? > > > > > > > > > > > > > > > > > > > > The fact that MPI API symbols exist in libpetsc.a makes PETSc not > > > > > > > really pure plant. Its still a plant/animal - so the goal of this > > > > > > > code > > > > > > > reorganization is still not met. > > > > > > > > > > > > As I said in my previous email; only Matt's plan is perfect. > > > > > > > > > > My argument is - the change mode is still imperfect - but at the cost > > > > > of breaking some user codes. [so its not worth the cost]. > > > > > > > > > > > > > > > > > > > [I might not have raised this issue earlier wrt merging mpiuni > > > > > > > completely into petsc - thats my omission sorry about that.] > > > > > > > > > > > > > > > > > > > > > > > > > > > > I suspec the following usage in FACETS/UEDGE will break with this > > > > > > > change: > > > > > > > > > > > > > > If you have package-A doing the same type of thing for its > > > > > > > sequential > > > > > > > implementation - it will have its own mpi_comm_size() etc > > > > > > > internally. > > > > > > > With this - mixing package-A with petsc-sequential will cause > > > > > > > duplicate symbol conflict. > > > > > > > > > > > > The fix is to not have any symbols in MPIUNI that are called MPI_xxx > > > > > > This is > > > > > > handled in mpiunis mpi.h with > > > > > > #if defined(MPIUNI_AVOID_MPI_NAMESPACE) > > > > > > we should just ALWAYS avoid the MPI Namespace > > > > > > > > > > > > > > > > > > > > > > > > > > The way I'm currently resolving this issue is: Only one package > > > > > > > should > > > > > > > provide [internal] MPI - the other package is compiled with > > > > > > > --with-mpi=enabled. > > > > > > > > > > > > > > I.e PETSc compiled with MPIUNI [says MPI_ENABLED]. Package-A is > > > > > > > now > > > > > > > compiled with MPI-ENABLED - but links with PETSc - that provides > > > > > > > MPIUNI - as MPI. > > > > > > > > > > > > > > Also due to this - we added more stuff to MPUNI to cover all MPI > > > > > > > symbol usage from FACETS. > > > > > > > > > > > > > > > > > > > > > So - the previous MPIUNI implementation scheme eventhough slightly > > > > > > > inconsistant, provided good user interface - with minimal > > > > > > > maintainance > > > > > > > overhead.. Matt's schem makes everything consistant - but with > > > > > > > extra > > > > > > > maintainance overhead. > > > > > > > > > > > > > > > > > > > We can certainly go back to the way it was if we need to. But you > > > > > > have > > > > > > not > > > > > > demonstrated that need, you've only speculated. > > > > > > Here is how it should work ok. > > > > > > 1) Several packages each use their own fake MPI, everything is fine > > > > > > unless two > > > > > > different mpi.h's get included but that should not happen. > > > > > > > > > > One of the ways it can conflict is: > > > > > > > > > > 'include' package1/mpif.h' > > > > > 'include mpiuni/mpif.h > > > > > > > > > > and there will be duplicate variable conflicts for constants > > > > > like MPI_COMM_WORLD, and others. > > > > > > > > > > I'll admit the old way of using mpiuni with other packages does not > > > > > work with all packages. [only with a select few codes that facets > > > > > folks tested with]. > > > > > > > > > > > > > > > > 2) Several packages each use PETSc's fake MPI. To support this we > > > > > > only > > > > > > need to > > > > > > have them add -I$PETSC_DIR/include/mpiuni in their compiles > > > > > > > > > > In this case - the user should be consious that he needs mpiuni from > > > > > user code - and enable another petsc build option? [or create > > > > > unportable makefile]. One more additional cost to the current > > > > > imperfect change. > > > > > > > > > > So I still prefer the previous scheme. > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > So I still prefer previous secheme. If that not acceptable - then > > > > > > > I > > > > > > > guess we have to go with Matt's scheme [ slplit up mpiuni into a > > > > > > > different package, add build/make support to it - and deal with > > > > > > > all > > > > > > > the petsc-maint that creates..]. But the current change really > > > > > > > breaks > > > > > > > things - so we should not do this. > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > On Thu, 25 Feb 2010, Barry Smith wrote: > > > > > > > > > > > > > > > > > > > > > > > After listening to our discussion of the half-plant/half animal > > > > > > > > handling > > > > > > > > of > > > > > > > > MPIUni I have adopted the plant :-) model. > > > > > > > > > > > > > > > > Background: All the packages in PETSc/packages and > > > > > > > > BuildSystem/config/packages > > > > > > > > have the following paradigm except MPI and BlasLapack. > > > > > > > > > > > > > > > > 1) PETSc can be built --with-package or --with-package=0 > > > > > > > > 2) If --with-package=0 then there is no PETSC_HAVE_package > > > > > > > > defined, > > > > > > > > no > > > > > > > > extra > > > > > > > > libraries to link against and no extra include paths to look in > > > > > > > > > > > > > > > > BlasLapack breaks this paradigm in only one way! You cannot use > > > > > > > > --with-blaspack=0 > > > > > > > > > > > > > > > > MPI breaks the paradigm more completely. You can use > > > > > > > > --with-mpi=0 > > > > > > > > BUT if > > > > > > > > you > > > > > > > > use --with-mpi=0 then PETSC_HAVE_MPI is STILL defined!!!!!! > > > > > > > > There is > > > > > > > > an > > > > > > > > extra > > > > > > > > library to link against and an extra include path to look in. > > > > > > > > > > > > > > > > The two possible solutions to resolve this perverse beast are > > > > > > > > 1) make mpiuni be a --download-mpiuni replacement for MPI, as we > > > > > > > > do > > > > > > > > --download-c-blaslapack (this would replace the current > > > > > > > > --with-mpi=0 > > > > > > > > support). > > > > > > > > 2) better supporting --with-mpi=0 without breaking the paradigm > > > > > > > > > > > > > > > > I agree with Matt that 1 is the more elegant solution, since it > > > > > > > > fits > > > > > > > > the > > > > > > > > paradigm perfectly. But having --with-mpi=0 is easier and more > > > > > > > > understandable > > > > > > > > to the user then explaining about downloading a dummy MPI. > > > > > > > > > > > > > > > > Thus I have implemented and pushed 2). When you use --with-mpi=0 > > > > > > > > 1) the macro PETSC_HAVE_MPI is not set > > > > > > > > 2) the list of include directories is not added to > > > > > > > > 3) the list of linked against libraries is not added to. > > > > > > > > > > > > > > > > I have implemented 2) and 3) by having in petsc.h (fortran also) > > > > > > > > #if defined(PETSC_HAVE_MPI) > > > > > > > > #include "mpi.h" > > > > > > > > #else > > > > > > > > #include "mpiuni/mpi.h" > > > > > > > > #endif > > > > > > > > and putting the dummy MPI stubs always into the PETSc libraries > > > > > > > > for > > > > > > > > both > > > > > > > > single library and multiple library PETSc installs. > > > > > > > > > > > > > > > > Note: this means one cannot have an include "mpi.h" in the uni > > > > > > > > case > > > > > > > > which > > > > > > > > bothered me initially but then Lisandro convinced me it was not > > > > > > > > a > > > > > > > > bad > > > > > > > > thing. > > > > > > > > > > > > > > > > The actual code changes to implement this were, of course, tiny. > > > > > > > > It > > > > > > > > is not > > > > > > > > perfect (only --download-mpiuni would be perfect :-), but it is > > > > > > > > better > > > > > > > > than > > > > > > > > before. > > > > > > > > > > > > > > > > > > > > > > > > Sorry Matt, > > > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
