Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jeff Hammond]

On Sat, Feb 10, 2018 at 9:29 AM Éric Chamberland <
eric.chamberl...@giref.ulaval.ca> wrote:

> Hi Matthew,
>
> Yes, I heard that the MPI C++ API has been deprecated.
>

It was *deprecated* in 2.2 and *deleted* in 3.0. Implementations may not
delete the code but there is absolutely no expectation that it is supported
by any recent implementation.

I fought the MPI Forum until the bitter end about this, by the way, because
we broke existing code in order to save the authors of new MPI
functionality a few minutes of LaTeX boilerplate generation.

It would be almost trivial to extract C++ bindings from MPICH and/or
Open-MPI as a standalone project to fix these issues, but nobody has made
the effort.

Jeff

Yes, the mpi_cxx is now missing.  Our link line is formed mainly with
> PETSC_WITH_EXTERNAL_LIB variable that is now:
>
> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib
> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib
> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64
> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64
> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64
> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4
> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc
> -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml
> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
> -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
> -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis
> -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm
> -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm
> -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++
> -ldl
>
> We add some stuff to this (our own compiled libs), but nothing related to
> MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our
> diffrement environments (think about different petsc version, MPI libs and
> compilers: clang, icc, g++) and it used to work until yesterday changes...
>
> As you can see in the diff, this "block" of libraries have been removed
> from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>
> --ldl
> --lmpi_cxx
> --lmpi
> --lstdc++
> --lm
> --lgcc_s
> --lpthread
>
> and into this block there was the "-lmpi_cxx" that we need...
>
> I could send you our whole line of link, but the error is into this small
> change introduced yesterday into master...
>
> Thanks a lot!
>
> Eric
>
> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>
> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland <
> eric.chamberl...@giref.ulaval.ca> wrote:
>
>> Hi,
>>
>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB
>> variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything
>> was fine until this night.
>>
>> It seems some libs have vanished from this variable, see the diff here:
>>
>>  -lptscotcherr
>>  -lesmumps
>>  -lscotch
>>  -lscotcherr
>>  -lm
>>  -lX11
>> +-lstdc++
>>  -ldl
>>  -lmpi_usempi
>>  -lmpi_mpifh
>>  -lmpi
>>  -lgfortran
>>  -lm
>>  -lgfortran
>>  -lm
>>  -lgcc_s
>>  -lquadmath
>>  -lpthread
>> --ldl
>> --lmpi_cxx
>> --lmpi
>> --lstdc++
>> --lm
>> --lgcc_s
>> --lpthread
>>  -lrt
>>  -lm
>>  -lpthread
>>  -lz
>> +-lstdc++
>>  -ldl
>>
>>
>> causing these errors at link phase for us:
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
>> const&), bool)':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
>> reference to `ompi_mpi_cxx_op_intercept' 
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
>> bool) const':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
>> reference to `MPI::Comm::Comm()' 
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intercomm::Merge(bool) const':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
>> undefined reference to `MPI::Comm::Comm()' 
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intracomm::Split(int, int) const':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
>> undefined reference to 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Le 18-02-10 à 16:49, Jed Brown a écrit :

Éric Chamberland  writes:


oh my, looks like "-lpetsc -lmpi" works for us!!!

(sounds of head banging on desk)

This should work any time libpetsc is a shared library.  You'll need the
whole grotesque link line if you need to link statically.
Doh!  ok, thanks for the specification because we need also a static 
link for the code coverage to work...


and for my peaceful sleep:

Since when it is possible to link with PETSC with just "-lpetsc -lmpi"?

We linked with our own "complicated" line since... about 1998 

Thanks,

Eric

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Matthew Knepley  writes:

> If we were in charge of all dependencies, that would be true. However,
> packages can dump in things that need to link with. if we could
> require that all dependencies be specified through our system, then we
> could topologically sort and put out all the link libs. This is
> exactly what we do for configure tests already.

What's the problem with doing that now?  A node in the graph can
correspond to more than one -l flag.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

oh my, looks like "-lpetsc -lmpi" works for us!!!

(sounds of head banging on desk)

;)

thanks!!!

Eric



Le 18-02-10 à 15:29, Éric Chamberland a écrit :

oh my god, if

-lpetsc -lmpi

works for me you make my day!!! :) really!

I will try it for sure!!!
I tried to maintain a very long linking line since sooo many years, if 
I can get ride of it, I will be happy!


Thanks,

Eric


Le 18-02-10 à 13:44, Jed Brown a écrit :

Éric Chamberland  writes:


Le 18-02-10 à 12:38, Jed Brown a écrit :

Éric Chamberland  writes:


and into this block there was the "-lmpi_cxx" that we need...

The point is that if you are linking C++ code that calls the MPI C++
interface, then *you* should link with mpicxx or equivalent.

The funny thing, is that we are hopefully *not* using the C++ API of
MPI.  We do use the C API since MPI 1.0.
Then, I am asking myself why this link error shows up now...since
nothing calls it... hmmm, let me dig into this...

Is StatistiqueMemoire not your code?

| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Op::Init(void (*)(void const*, void*, int,
| MPI::Datatype const&), bool)':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: 
undefined reference to `ompi_mpi_cxx_op_intercept'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Intracomm::Create_graph(int, int const*, int const*,
| bool) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Intercomm::Merge(bool) const':
| 
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Intracomm::Split(int, int) const':
| 
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Intracomm::Create(MPI::Group const&) const':
| 
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:

| In function `MPI::Intracomm::Clone() const':


You should not depend on PETSc to provide anything but PETSc to your
application (so if you call other libraries that your configuration of
PETSc uses, you should take responsibility to link them explicitly; 
this

Wait: you mean for example if I configure MUMPS (or not) with PETSc, I
have to build my link line with all MUMPS dependencies or not myself?

Do you call MUMPS directly or through the PETSc interface?

Think about this from the perspective of packaging with shared
libraries, where we ask what needs to be updated when interfaces change.
Suppose we have these interface dependencies:

   libpetsc : libmumps libmpi
   app : libpetsc libmpi

Since MUMPS makes no guarantee of binary compatibility between releases,
updating MUMPS 5.1.2 to 5.1.3 requires rebuilding everything that link
to it.  Since PETSc calls MUMPS directly, libpetsc must be rebuilt.  If
your App does not call MUMPS directly, then you should link it with

   -lpetsc -lmpi

Now your binaries continue to work after libpetsc has been updated to
link the newer libmumps.  If you needlessly linked libmumps without
calling it directly, then you would also need to rebuild your App.  That
is called overlinking and is prohibited by most packaging guidelines
because it wastes lots of time and bandwidth for maintainers and users.

If you use pkg-config with PETSc, you get only -lpetsc when linking with
shared libraries.  (Pkg-config will give you everything for static
linking because it is needed and the concept of overlinking doesn't
exist in the same sense, though removing excess from the link line is
still desirable to make it easier to read and faster to link.)

Note that PETSc's own dependency management is messy because someone in
the early days of BuildSystem thought that concatenating strings was
sufficient, instead of maintaining a structured dependency graph to be
topologically sorted at the final stage (pkg-config does this).


I wanted to be "lazy" and to use the same line *you* are using for
passing libs (and in the good *order* which is not easy to 
manage)... as

I can rely on this line to link "pure" petsc examples... ;)

is important when using shared libraries).  But you should definitely
not depend on PETSc to provide your application with stuff that has 
been
REMOVED from 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jeff Hammond]

On Sat, Feb 10, 2018 at 1:48 PM, Jed Brown  wrote:

> Éric Chamberland  writes:
>
> > ok, let put our in-house code away, simply try this simple example:
> >
> > --
> >
> > #include 
> >
> > int main() {
> > /*empty!! let's see the .o */
> >
> > return 1;
> >
> > }
> > -
> >
> > then:
> >
> > g++ -c test.cc -I/opt/openmpi-1.10.2/include -o test.o
> >
> > nm -C test.o | grep "MPI::" | grep " U "
> >
> >   U MPI::Win::Free()
> >   U MPI::Comm::Comm()
> >   U MPI::Datatype::Free()
> >
> > so: as soon you are compiling a c++ file with this openmpi 1.10.2 (even
> > if you use gcc -c test.cc -I/opt/openmpi-1.10.2/include -o test.o in my
> > example) it *requires* you to link with mpi_cxx lib...
>
> That's sad, but is this not avoided by defining OMPI_SKIP_MPICXX before
> including mpi.h?  (There is also MPICH_SKIP_MPICXX if you want.)  This
> is done by default in petscsys.h, but you might not include it in that
> file or don't include it before mpi.h is included.
>

Indeed, everybody should compiler MPI codes with "-DMPICH_SKIP_MPICXX=1
-DOMPI_SKIP_MPICXX=1" now.

I'll ask MPICH and Open-MPI to switch the default to exclude C++ bindings.

Jeff

-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

oh my god, if

-lpetsc -lmpi

works for me you make my day!!! :) really!

I will try it for sure!!!
I tried to maintain a very long linking line since sooo many years, if I 
can get ride of it, I will be happy!


Thanks,

Eric


Le 18-02-10 à 13:44, Jed Brown a écrit :

Éric Chamberland  writes:


Le 18-02-10 à 12:38, Jed Brown a écrit :

Éric Chamberland  writes:


and into this block there was the "-lmpi_cxx" that we need...

The point is that if you are linking C++ code that calls the MPI C++
interface, then *you* should link with mpicxx or equivalent.

The funny thing, is that we are hopefully *not* using the C++ API of
MPI.  We do use the C API since MPI 1.0.
Then, I am asking myself why this link error shows up now...since
nothing calls it... hmmm, let me dig into this...

Is StatistiqueMemoire not your code?

| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Op::Init(void (*)(void const*, void*, int,
| MPI::Datatype const&), bool)':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
reference to `ompi_mpi_cxx_op_intercept'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Intracomm::Create_graph(int, int const*, int const*,
| bool) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Intercomm::Merge(bool) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Intracomm::Split(int, int) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Intracomm::Create(MPI::Group const&) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
| In function `MPI::Intracomm::Clone() const':


You should not depend on PETSc to provide anything but PETSc to your
application (so if you call other libraries that your configuration of
PETSc uses, you should take responsibility to link them explicitly; this

Wait: you mean for example if I configure MUMPS (or not) with PETSc, I
have to build my link line with all MUMPS dependencies or not myself?

Do you call MUMPS directly or through the PETSc interface?

Think about this from the perspective of packaging with shared
libraries, where we ask what needs to be updated when interfaces change.
Suppose we have these interface dependencies:

   libpetsc : libmumps libmpi
   app : libpetsc libmpi

Since MUMPS makes no guarantee of binary compatibility between releases,
updating MUMPS 5.1.2 to 5.1.3 requires rebuilding everything that link
to it.  Since PETSc calls MUMPS directly, libpetsc must be rebuilt.  If
your App does not call MUMPS directly, then you should link it with

   -lpetsc -lmpi

Now your binaries continue to work after libpetsc has been updated to
link the newer libmumps.  If you needlessly linked libmumps without
calling it directly, then you would also need to rebuild your App.  That
is called overlinking and is prohibited by most packaging guidelines
because it wastes lots of time and bandwidth for maintainers and users.

If you use pkg-config with PETSc, you get only -lpetsc when linking with
shared libraries.  (Pkg-config will give you everything for static
linking because it is needed and the concept of overlinking doesn't
exist in the same sense, though removing excess from the link line is
still desirable to make it easier to read and faster to link.)

Note that PETSc's own dependency management is messy because someone in
the early days of BuildSystem thought that concatenating strings was
sufficient, instead of maintaining a structured dependency graph to be
topologically sorted at the final stage (pkg-config does this).


I wanted to be "lazy" and to use the same line *you* are using for
passing libs (and in the good *order* which is not easy to manage)... as
I can rely on this line to link "pure" petsc examples... ;)

is important when using shared libraries).  But you should definitely
not depend on PETSc to provide your application with stuff that has been
REMOVED from MPI (more than five years ago) and that PETSc does not use.

I agree, this C++ MPI API is not a very good excuse to add boring work
into any software configuration and 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

  Eric,

Ok thanks for the update, and for not requiring us to undo recent 
"improvements" to our handling of compiler libraries.

   Barry


> On Feb 10, 2018, at 2:25 PM, Éric Chamberland 
>  wrote:
> 
> ok, let put our in-house code away, simply try this simple example:
> 
> --
> 
> #include 
> 
> int main() {
> /*empty!! let's see the .o */
> 
> return 1;
> 
> }
> -
> 
> then:
> 
> g++ -c test.cc -I/opt/openmpi-1.10.2/include -o test.o
> 
> nm -C test.o | grep "MPI::" | grep " U "
> 
>  U MPI::Win::Free()
>  U MPI::Comm::Comm()
>  U MPI::Datatype::Free()
> 
> so: as soon you are compiling a c++ file with this openmpi 1.10.2 (even if 
> you use gcc -c test.cc -I/opt/openmpi-1.10.2/include -o test.o in my example) 
> it *requires* you to link with mpi_cxx lib...
> 
> That's the "problem", that's *my* fault to have relied on a PETSC variable to 
> link with an mpi c++ lib
> 
> So I will add the mpi_cxx lib to our makefiles for these older versions of 
> mpi...
> 
> Sorry for the annoyance...
> 
> But, as usual, thanks for your fast responses guys, you all deserve a gold 
> medal!!! :)
> 
> Eric
> 
> 
> Le 18-02-10 à 14:37, Jed Brown a écrit :
>> "Smith, Barry F."  writes:
>> 
>>>Then the testing is more involved, need to first check if C++ bindings 
>>> exist and then check if supplied automatically by compilers (like on Cray). 
>>> Annoying, I'd prefer to avoid but I still don't have the definitive word if 
>>> C++ bindings are the issue from Eric.
>> The undefined symbols are C++ bindings and they are referenced from a
>> source file named StatistiqueMemoire.* which is not part of PETSc or a
>> PETSc dependency.  We don't reference them so we shouldn't try to link
>> them.
> 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

ok, let put our in-house code away, simply try this simple example:

--

#include 

int main() {
/*empty!! let's see the .o */

return 1;

}
-

then:

g++ -c test.cc -I/opt/openmpi-1.10.2/include -o test.o

nm -C test.o | grep "MPI::" | grep " U "

 U MPI::Win::Free()
 U MPI::Comm::Comm()
 U MPI::Datatype::Free()

so: as soon you are compiling a c++ file with this openmpi 1.10.2 (even 
if you use gcc -c test.cc -I/opt/openmpi-1.10.2/include -o test.o in my 
example) it *requires* you to link with mpi_cxx lib...


That's the "problem", that's *my* fault to have relied on a PETSC 
variable to link with an mpi c++ lib


So I will add the mpi_cxx lib to our makefiles for these older versions 
of mpi...


Sorry for the annoyance...

But, as usual, thanks for your fast responses guys, you all deserve a 
gold medal!!! :)


Eric


Le 18-02-10 à 14:37, Jed Brown a écrit :

"Smith, Barry F."  writes:


Then the testing is more involved, need to first check if C++ bindings 
exist and then check if supplied automatically by compilers (like on Cray). 
Annoying, I'd prefer to avoid but I still don't have the definitive word if C++ 
bindings are the issue from Eric.

The undefined symbols are C++ bindings and they are referenced from a
source file named StatistiqueMemoire.* which is not part of PETSc or a
PETSc dependency.  We don't reference them so we shouldn't try to link
them.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

> ok, let put our in-house code away, simply try this simple example:
>
> --
>
> #include 
>
> int main() {
> /*empty!! let's see the .o */
>
> return 1;
>
> }
> -
>
> then:
>
> g++ -c test.cc -I/opt/openmpi-1.10.2/include -o test.o
>
> nm -C test.o | grep "MPI::" | grep " U "
>
>   U MPI::Win::Free()
>   U MPI::Comm::Comm()
>   U MPI::Datatype::Free()
>
> so: as soon you are compiling a c++ file with this openmpi 1.10.2 (even 
> if you use gcc -c test.cc -I/opt/openmpi-1.10.2/include -o test.o in my 
> example) it *requires* you to link with mpi_cxx lib...

That's sad, but is this not avoided by defining OMPI_SKIP_MPICXX before
including mpi.h?  (There is also MPICH_SKIP_MPICXX if you want.)  This
is done by default in petscsys.h, but you might not include it in that
file or don't include it before mpi.h is included.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

> oh my, looks like "-lpetsc -lmpi" works for us!!!
>
> (sounds of head banging on desk)

This should work any time libpetsc is a shared library.  You'll need the
whole grotesque link line if you need to link statically.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Matthew Knepley]

On Sat, Feb 10, 2018 at 2:34 PM, Jed Brown  wrote:

> Matthew Knepley  writes:
>
> >> Note that PETSc's own dependency management is messy because someone in
> >> the early days of BuildSystem thought that concatenating strings was
> >> sufficient, instead of maintaining a structured dependency graph to be
> >> topologically sorted at the final stage (pkg-config does this).
> >>
> >
> > Of course, we do have a structured dependency graph. Where is it not
> being
> > used?
>
> If the DAG structure was preserved, toStringNoDupes would never need to
> work with a list containing duplicate flags, including libraries like
> -ldl and -lm, because the topological sort would never have created
> duplicates in the first place.
>

If we were in charge of all dependencies, that would be true. However,
packages can
dump in things that need to link with. if we could require that all
dependencies be
specified through our system, then we could topologically sort and put out
all the
link libs. This is exactly what we do for configure tests already.

   Matt


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Le 18-02-10 à 17:44, Jeff Hammond a écrit :


Indeed, everybody should compiler MPI codes with 
"-DMPICH_SKIP_MPICXX=1 -DOMPI_SKIP_MPICXX=1" now.


I'll ask MPICH and Open-MPI to switch the default to exclude C++ bindings.


Will definitely switch my default compiling options right now.

And I agree that it is relevant to switch that by default for all MPI 
distributions.  Thanks for forwarding!


Eric


Jeff

--
Jeff Hammond
jeff.scie...@gmail.com 
http://jeffhammond.github.io/

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Le 18-02-10 à 16:48, Jed Brown a écrit :

That's sad, but is this not avoided by defining OMPI_SKIP_MPICXX before
including mpi.h?  (There is also MPICH_SKIP_MPICXX if you want.)  This
is done by default in petscsys.h, but you might not include it in that
file or don't include it before mpi.h is included.
Ok, didn't knew about this...  I think this is definitely a good 
solution!!! :)


Thanks again!

Eric

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

> Le 18-02-10 à 16:49, Jed Brown a écrit :
>> Éric Chamberland  writes:
>>
>>> oh my, looks like "-lpetsc -lmpi" works for us!!!
>>>
>>> (sounds of head banging on desk)
>> This should work any time libpetsc is a shared library.  You'll need the
>> whole grotesque link line if you need to link statically.
> Doh!  ok, thanks for the specification because we need also a static 
> link for the code coverage to work...

You can use pkg-config --static.

> and for my peaceful sleep:
>
> Since when it is possible to link with PETSC with just "-lpetsc -lmpi"?

Probably as long as --with-shared-libraries has existed.

[petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Hi,

we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB 
variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and 
everything was fine until this night.


It seems some libs have vanished from this variable, see the diff here:

 -lptscotcherr
 -lesmumps
 -lscotch
 -lscotcherr
 -lm
 -lX11
+-lstdc++
 -ldl
 -lmpi_usempi
 -lmpi_mpifh
 -lmpi
 -lgfortran
 -lm
 -lgfortran
 -lm
 -lgcc_s
 -lquadmath
 -lpthread
--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread
 -lrt
 -lm
 -lpthread
 -lz
+-lstdc++
 -ldl


causing these errors at link phase for us:

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Op::Init(void (*)(void const*, void*, int, 
MPI::Datatype const&), bool)':

/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
reference to `ompi_mpi_cxx_op_intercept'
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
bool) const':

/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
reference to `MPI::Comm::Comm()'
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Intercomm::Merge(bool) const':

/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: undefined 
reference to `MPI::Comm::Comm()'
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Intracomm::Split(int, int) const':

/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: undefined 
reference to `MPI::Comm::Comm()'
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Intracomm::Create(MPI::Group const&) const':

/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: undefined 
reference to `MPI::Comm::Comm()'
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: 
In function `MPI::Intracomm::Clone() const':


Is this a normal and definitive change or an unwanted/unobserved bug?

Thanks,

Eric

ps: here are the logs:

this night:

-

http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.log
http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log


a day before:


http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.log
http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Hi Matthew,

Yes, I heard that the MPI C++ API has been deprecated.

Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
PETSC_WITH_EXTERNAL_LIB variable that is now:


PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
-Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
-L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
-Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
-L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
-L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
-L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc 
-lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps 
-lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml 
-lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd 
-lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread 
-lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis 
-lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm 
-lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm 
-lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz 
-lstdc++ -ldl


We add some stuff to this (our own compiled libs), but nothing related 
to MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all 
our diffrement environments (think about different petsc version, MPI 
libs and compilers: clang, icc, g++) and it used to work until yesterday 
changes...


As you can see in the diff, this "block" of libraries have been removed 
from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):


--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread

and into this block there was the "-lmpi_cxx" that we need...

I could send you our whole line of link, but the error is into this 
small change introduced yesterday into master...


Thanks a lot!

Eric

Le 18-02-10 à 10:34, Matthew Knepley a écrit :
On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland 
> wrote:


Hi,

we used to link our c++ code with PETSc using
PETSC_WITH_EXTERNAL_LIB variable defined in
$PETSC_DIR/lib/petsc/conf/petscvariables and everything was fine
until this night.

It seems some libs have vanished from this variable, see the diff
here:

  -lptscotcherr
  -lesmumps
  -lscotch
  -lscotcherr
  -lm
  -lX11
+-lstdc++
  -ldl
  -lmpi_usempi
  -lmpi_mpifh
  -lmpi
  -lgfortran
  -lm
  -lgfortran
  -lm
  -lgcc_s
  -lquadmath
  -lpthread
--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread
  -lrt
  -lm
  -lpthread
  -lz
+-lstdc++
  -ldl


causing these errors at link phase for us:


/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Op::Init(void (*)(void const*, void*, int,
MPI::Datatype const&), bool)':
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
reference to `ompi_mpi_cxx_op_intercept'

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Intracomm::Create_graph(int, int const*, int
const*, bool) const':
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
reference to `MPI::Comm::Comm()'

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Intercomm::Merge(bool) const':
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Intracomm::Split(int, int) const':
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Intracomm::Create(MPI::Group const&) const':
/opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'

/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
In function `MPI::Intracomm::Clone() const':

Hi Eric,

These symbols are all coming from -lmpi_cxx. I would note that I 
believe the MPI Forum has deprecated the C++ interface, so 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Matthew Knepley]

On Sat, Feb 10, 2018 at 12:29 PM, Éric Chamberland <
eric.chamberl...@giref.ulaval.ca> wrote:

> Hi Matthew,
>
> Yes, I heard that the MPI C++ API has been deprecated.
>
> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with
> PETSC_WITH_EXTERNAL_LIB variable that is now:
>
> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib
> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib
> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/intel/
> composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/openmpi-1.10.2/lib
> -L/opt/openmpi-1.10.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -Wl,-rpath,/opt/intel/
> composer_xe_2015.2.164/compiler/lib/intel64 
> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64
> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 -Wl,-rpath,/opt/intel/
> composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4
> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc
> -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml
> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
> -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
> -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis
> -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm
> -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm
> -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++
> -ldl
>
> We add some stuff to this (our own compiled libs), but nothing related to
> MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our
> diffrement environments (think about different petsc version, MPI libs and
> compilers: clang, icc, g++) and it used to work until yesterday changes...
>
> As you can see in the diff, this "block" of libraries have been removed
> from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>
> --ldl
> --lmpi_cxx
> --lmpi
> --lstdc++
> --lm
> --lgcc_s
> --lpthread
>
> and into this block there was the "-lmpi_cxx" that we need...
>

These libraries were removed from the definition because they were
redundant if you are using the compilers that PETSc was configured with:

Compilers:
  C Compiler: /opt/openmpi-1.10.2/bin/mpicc  -fPIC  -Wall
-Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector
-g3
  C++ Compiler:   /opt/openmpi-1.10.2/bin/mpicxx  -Wall -Wwrite-strings
-Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g   -fPIC
  Fortran Compiler:   /opt/openmpi-1.10.2/bin/mpif90  -fPIC -Wall
-ffree-line-length-0 -Wno-unused-dummy-argument -g

The C++ compiler /opt/openmpi-1.10.2/bin/mpicxx does link using -lmpi_cxx,
but your link must not be using it. That is why I wanted to see the whole
link line.

  Thanks,

 Matt

I could send you our whole line of link, but the error is into this small
> change introduced yesterday into master...
>
> Thanks a lot!
>
> Eric
>
> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>
> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland  ulaval.ca> wrote:
>
>> Hi,
>>
>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB
>> variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and
>> everything was fine until this night.
>>
>> It seems some libs have vanished from this variable, see the diff here:
>>
>>  -lptscotcherr
>>  -lesmumps
>>  -lscotch
>>  -lscotcherr
>>  -lm
>>  -lX11
>> +-lstdc++
>>  -ldl
>>  -lmpi_usempi
>>  -lmpi_mpifh
>>  -lmpi
>>  -lgfortran
>>  -lm
>>  -lgfortran
>>  -lm
>>  -lgcc_s
>>  -lquadmath
>>  -lpthread
>> --ldl
>> --lmpi_cxx
>> --lmpi
>> --lstdc++
>> --lm
>> --lgcc_s
>> --lpthread
>>  -lrt
>>  -lm
>>  -lpthread
>>  -lz
>> +-lstdc++
>>  -ldl
>>
>>
>> causing these errors at link phase for us:
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
>> const&), bool)':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
>> reference to `ompi_mpi_cxx_op_intercept' 
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
>> bool) const':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
>> reference to `MPI::Comm::Comm()' 
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intercomm::Merge(bool) const':
>> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
>> undefined reference to `MPI::Comm::Comm()' 
>> 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

> Hi Matthew,
>
> Yes, I heard that the MPI C++ API has been deprecated.
>
> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
> PETSC_WITH_EXTERNAL_LIB variable that is now:
>
> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc 
> -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps 
> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml 
> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd 
> -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread 
> -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis 
> -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm 
> -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm 
> -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz 
> -lstdc++ -ldl
>
> We add some stuff to this (our own compiled libs), but nothing related 
> to MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all 
> our diffrement environments (think about different petsc version, MPI 
> libs and compilers: clang, icc, g++) and it used to work until yesterday 
> changes...
>
> As you can see in the diff, this "block" of libraries have been removed 
> from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>
> --ldl
> --lmpi_cxx
> --lmpi
> --lstdc++
> --lm
> --lgcc_s
> --lpthread
>
> and into this block there was the "-lmpi_cxx" that we need...

The point is that if you are linking C++ code that calls the MPI C++
interface, then *you* should link with mpicxx or equivalent.

You should not depend on PETSc to provide anything but PETSc to your
application (so if you call other libraries that your configuration of
PETSc uses, you should take responsibility to link them explicitly; this
is important when using shared libraries).  But you should definitely
not depend on PETSc to provide your application with stuff that has been
REMOVED from MPI (more than five years ago) and that PETSc does not use.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

  Eric,

   We need the entire link line.

   What linker are you using C, or C++?  This is important.

   Do you have dependencies on MPI C++ symbols? In other words, are you using 
C++ MPI bindings?

I cannot explain why all the libraries you listed would disappear from 
PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass under 
some circumstances less system (including MPI libraries) explicitly since they 
are already usually passed by the linker. This may be causing your difficulties.

Eagerly awaiting your reply.

 Barry


> On Feb 10, 2018, at 11:29 AM, Éric Chamberland 
>  wrote:
> 
> Hi Matthew,
> 
> Yes, I heard that the MPI C++ API has been deprecated.
> 
> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
> PETSC_WITH_EXTERNAL_LIB variable that is now:
> 
> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc -lsuperlu 
> -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common 
> -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu 
> -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig 
> -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 
> -liomp5 -ldl -lpthread -lparmetis -lmetis -lptesmumps -lptscotch 
> -lptscotcherr -lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl 
> -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s 
> -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ -ldl
> We add some stuff to this (our own compiled libs), but nothing related to MPI 
> or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our 
> diffrement environments (think about different petsc version, MPI libs and 
> compilers: clang, icc, g++) and it used to work until yesterday changes...
> 
> As you can see in the diff, this "block" of libraries have been removed from 
> PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
> --ldl
> --lmpi_cxx
> --lmpi
> --lstdc++
> --lm
> --lgcc_s
> --lpthread
> 
> and into this block there was the "-lmpi_cxx" that we need...
> 
> I could send you our whole line of link, but the error is into this small 
> change introduced yesterday into master...
> 
> Thanks a lot!
> 
> Eric
> 
> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland 
>>  wrote:
>> Hi,
>> 
>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB 
>> variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything 
>> was fine until this night.
>> 
>> It seems some libs have vanished from this variable, see the diff here:
>> 
>>  -lptscotcherr
>>  -lesmumps
>>  -lscotch
>>  -lscotcherr
>>  -lm
>>  -lX11
>> +-lstdc++
>>  -ldl
>>  -lmpi_usempi
>>  -lmpi_mpifh
>>  -lmpi
>>  -lgfortran
>>  -lm
>>  -lgfortran
>>  -lm
>>  -lgcc_s
>>  -lquadmath
>>  -lpthread
>> --ldl
>> --lmpi_cxx
>> --lmpi
>> --lstdc++
>> --lm
>> --lgcc_s
>> --lpthread
>>  -lrt
>>  -lm
>>  -lpthread
>>  -lz
>> +-lstdc++
>>  -ldl
>> 
>> 
>> causing these errors at link phase for us:
>> 
>> 
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
>> const&), bool)': 
>> 
>> /opt/openmpi-1.10.2/include/
>> openmpi/ompi/mpi/cxx/op_inln.
>> h:122: undefined reference to `ompi_mpi_cxx_op_intercept'
>> 
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
>> bool) const': 
>> 
>> /opt/openmpi-1.10.2/include/
>> openmpi/ompi/mpi/cxx/
>> intracomm.h:25: undefined reference to `MPI::Comm::Comm()'
>> 
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function `MPI::Intercomm::Merge(bool) const': 
>> 
>> /opt/openmpi-1.10.2/include/
>> openmpi/ompi/mpi/cxx/
>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>> 
>>  
>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>  In function 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

  Eric

  Is your main program C, C++ or Fortran?



> On Feb 10, 2018, at 11:37 AM, Smith, Barry F.  wrote:
> 
> 
>  Eric,
> 
>   We need the entire link line.
> 
>   What linker are you using C, or C++?  This is important.
> 
>   Do you have dependencies on MPI C++ symbols? In other words, are you using 
> C++ MPI bindings?
> 
>I cannot explain why all the libraries you listed would disappear from 
> PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass 
> under some circumstances less system (including MPI libraries) explicitly 
> since they are already usually passed by the linker. This may be causing your 
> difficulties.
> 
>Eagerly awaiting your reply.
> 
> Barry
> 
> 
>> On Feb 10, 2018, at 11:29 AM, Éric Chamberland 
>>  wrote:
>> 
>> Hi Matthew,
>> 
>> Yes, I heard that the MPI C++ API has been deprecated.
>> 
>> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
>> PETSC_WITH_EXTERNAL_LIB variable that is now:
>> 
>> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
>> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
>> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
>> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc -lsuperlu 
>> -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common 
>> -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu 
>> -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig 
>> -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 
>> -liomp5 -ldl -lpthread -lparmetis -lmetis -lptesmumps -lptscotch 
>> -lptscotcherr -lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl 
>> -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s 
>> -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ -ldl
>> We add some stuff to this (our own compiled libs), but nothing related to 
>> MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our 
>> diffrement environments (think about different petsc version, MPI libs and 
>> compilers: clang, icc, g++) and it used to work until yesterday changes...
>> 
>> As you can see in the diff, this "block" of libraries have been removed from 
>> PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>> --ldl
>> --lmpi_cxx
>> --lmpi
>> --lstdc++
>> --lm
>> --lgcc_s
>> --lpthread
>> 
>> and into this block there was the "-lmpi_cxx" that we need...
>> 
>> I could send you our whole line of link, but the error is into this small 
>> change introduced yesterday into master...
>> 
>> Thanks a lot!
>> 
>> Eric
>> 
>> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>>> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland 
>>>  wrote:
>>> Hi,
>>> 
>>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB 
>>> variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything 
>>> was fine until this night.
>>> 
>>> It seems some libs have vanished from this variable, see the diff here:
>>> 
>>> -lptscotcherr
>>> -lesmumps
>>> -lscotch
>>> -lscotcherr
>>> -lm
>>> -lX11
>>> +-lstdc++
>>> -ldl
>>> -lmpi_usempi
>>> -lmpi_mpifh
>>> -lmpi
>>> -lgfortran
>>> -lm
>>> -lgfortran
>>> -lm
>>> -lgcc_s
>>> -lquadmath
>>> -lpthread
>>> --ldl
>>> --lmpi_cxx
>>> --lmpi
>>> --lstdc++
>>> --lm
>>> --lgcc_s
>>> --lpthread
>>> -lrt
>>> -lm
>>> -lpthread
>>> -lz
>>> +-lstdc++
>>> -ldl
>>> 
>>> 
>>> causing these errors at link phase for us:
>>> 
>>> 
>>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>>  In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
>>> const&), bool)': 
>>> 
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/op_inln.
>>> h:122: undefined reference to `ompi_mpi_cxx_op_intercept'
>>> 
>>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>>  In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
>>> bool) const': 
>>> 
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/
>>> intracomm.h:25: undefined reference to `MPI::Comm::Comm()'
>>> 
>>> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>>>  In function `MPI::Intercomm::Merge(bool) const': 
>>> 
>>> 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

"Smith, Barry F."  writes:

>   Eric,
>
>We need the entire link line.
>
>What linker are you using C, or C++?  This is important.
>
>Do you have dependencies on MPI C++ symbols? In other words, are you using 
> C++ MPI bindings?
>
> I cannot explain why all the libraries you listed would disappear from 
> PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass 
> under some circumstances less system (including MPI libraries) explicitly 
> since they are already usually passed by the linker. This may be causing your 
> difficulties.

According to Eric's output, -lstdc++ appears twice now, -lmpi still
appears, -ldl still appears (twice), -lm still appears (4 times),
-lpthread still appears (twice), and -lgcc_s still appears.  So the only
library actually eliminated is -lmpi_cxx which is provided by the mpicxx
(despite having been removed from the standard in 2012).

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Ok guys!

You found it (again...).  We are no using mpic++ to link!!!

We are not always using the same compiler that we used to compile 
petsc+mpi, so we do not use mpic++ to compile (excepted for intelmpi).


For example, I compile MPI+Petsc with g++, but I am working with clang++ 
and may switch to g++ for my day to day work.  This is perfectly working 
since... long long time ago (I won't reveal my age... ;) )


I understand you choice, but your PETSC_WITH_EXTERNAL_LIB variable was 
helping me to painlessly find the good linking options.


I can work to manually add it when it is relevant into our compilation 
flags...


How did you managed to extract the libs from all the different flavors 
of MPI (mpich, openmpi, intelmpi) so I can now reproduce what you did 
before?


(maybe this should have been my question... )

Thanks again,

Eric

Le 18-02-10 à 12:37, Smith, Barry F. a écrit :

   Eric,

We need the entire link line.

What linker are you using C, or C++?  This is important.

Do you have dependencies on MPI C++ symbols? In other words, are you using 
C++ MPI bindings?

 I cannot explain why all the libraries you listed would disappear from 
PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass under 
some circumstances less system (including MPI libraries) explicitly since they 
are already usually passed by the linker. This may be causing your difficulties.

 Eagerly awaiting your reply.

  Barry



On Feb 10, 2018, at 11:29 AM, Éric Chamberland 
 wrote:

Hi Matthew,

Yes, I heard that the MPI C++ API has been deprecated.

Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
PETSC_WITH_EXTERNAL_LIB variable that is now:

PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
-Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
-L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
-Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
-L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
-L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
-Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
-L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc -lsuperlu 
-lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common 
-lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu 
-lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig 
-lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 
-liomp5 -ldl -lpthread -lparmetis -lmetis -lptesmumps -lptscotch -lptscotcherr 
-lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh 
-lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm 
-lpthread -lz -lstdc++ -ldl
We add some stuff to this (our own compiled libs), but nothing related to MPI 
or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our 
diffrement environments (think about different petsc version, MPI libs and 
compilers: clang, icc, g++) and it used to work until yesterday changes...

As you can see in the diff, this "block" of libraries have been removed from 
PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread

and into this block there was the "-lmpi_cxx" that we need...

I could send you our whole line of link, but the error is into this small 
change introduced yesterday into master...

Thanks a lot!

Eric

Le 18-02-10 à 10:34, Matthew Knepley a écrit :

On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland 
 wrote:
Hi,

we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB variable 
defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything was fine 
until this night.

It seems some libs have vanished from this variable, see the diff here:

  -lptscotcherr
  -lesmumps
  -lscotch
  -lscotcherr
  -lm
  -lX11
+-lstdc++
  -ldl
  -lmpi_usempi
  -lmpi_mpifh
  -lmpi
  -lgfortran
  -lm
  -lgfortran
  -lm
  -lgcc_s
  -lquadmath
  -lpthread
--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread
  -lrt
  -lm
  -lpthread
  -lz
+-lstdc++
  -ldl


causing these errors at link phase for us:


  
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
const&), bool)':

/opt/openmpi-1.10.2/include/
openmpi/ompi/mpi/cxx/op_inln.
h:122: undefined reference to `ompi_mpi_cxx_op_intercept'

  
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 In function 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

   Eric,

 Please send configure.log

> On Feb 10, 2018, at 11:44 AM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>  Eric,
>> 
>>   We need the entire link line.
>> 
>>   What linker are you using C, or C++?  This is important.
>> 
>>   Do you have dependencies on MPI C++ symbols? In other words, are you using 
>> C++ MPI bindings?
>> 
>>I cannot explain why all the libraries you listed would disappear from 
>> PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass 
>> under some circumstances less system (including MPI libraries) explicitly 
>> since they are already usually passed by the linker. This may be causing 
>> your difficulties.
> 
> According to Eric's output, -lstdc++ appears twice now, -lmpi still
> appears, -ldl still appears (twice), -lm still appears (4 times),
> -lpthread still appears (twice), and -lgcc_s still appears.  So the only
> library actually eliminated is-lmpi_cxx which is provided by the mpicxx
> (despite having been removed from the standard in 2012).

Jed,

   I did my best to eliminate unneeded library links, it is not trivial as 
you seem to believe it is to get just exactly the libraries listed that are 
needed. 

   I know how to get the -lmpi_cxx  listed again (I need to include a call 
to a MPI C++ function in the checkCXXlibraries) but this will result in a bunch 
more libraries being redundantly brought in.  Since Eric is the only person in 
the entire world who ever used the C++ bindings it would be nice if I didn't 
need to add the C++ function test and add additional redundant libraries you 
will complain about for many runs.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

ok, here are the logs from this night and yesterday:

this night:

-

http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.log
http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log


a day before:


http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.log
http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log


Eric


Le 18-02-10 à 12:51, Smith, Barry F. a écrit :

Eric,

  Please send configure.log

Re: [petsc-dev] MPI_Attr_get test fails

[Jed Brown]

Mark Adams  writes:

> On Fri, Feb 9, 2018 at 9:39 PM, Jeff Hammond  wrote:
>
>> https://msdn.microsoft.com/en-us/library/dn473234(v=vs.85).aspx
>>
>> This function name is archaic and removed in MPI-3. Use the new name.
>>
>>
> OK, Jed did this a few months ago.
>
> It still fails because my code uses PETSC_COMM_WORLD. I guess I need to
> change that.

This runs cleanly with MPICH and Open MPI.  Mark, PETSC_COMM_WORLD is
_usually_ equivalent to MPI_COMM_WORLD.  You should *never* use it in
PETSc library code.  It is not a "PETSc communicator" or "inner
communicator" -- those terms refer to a dup'd communicator held by PETSc
objects for PETSc's exclusive use.

#include 

int main(int argc, char **args)
{
  PetscErrorCode ierr;
  MPI_Comm   newcomm;
  ierr = PetscInitialize(, , NULL, NULL);CHKERRQ(ierr);
  ierr = PetscCommDuplicate(PETSC_COMM_WORLD, , NULL);CHKERRQ(ierr);
  { // debug
PetscCommCounter *counter;
PetscMPIInt  flg;
ierr = 
MPI_Comm_get_attr(newcomm,Petsc_Counter_keyval,,);CHKERRQ(ierr);
if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI 
communicator supplied?");
  }
  ierr = PetscCommDestroy();CHKERRQ(ierr);
  return PetscFinalize();
}

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

> On Feb 10, 2018, at 11:51 AM, Éric Chamberland 
>  wrote:
> 
> Ok guys!
> 
> You found it (again...).  We are no using mpic++ to link!!!
> 
> We are not always using the same compiler that we used to compile petsc+mpi, 
> so we do not use mpic++ to compile (excepted for intelmpi).
> 
> For example, I compile MPI+Petsc with g++, but I am working with clang++ and 
> may switch to g++ for my day to day work.  This is perfectly working since... 
> long long time ago (I won't reveal my age... ;) )

   WOW. You are lucky this worked for you. I don't think missing C++ 
compilers/linkers is ever safe?

> 
> I understand you choice, but your PETSC_WITH_EXTERNAL_LIB variable was 
> helping me to painlessly find the good linking options.
> 
> I can work to manually add it when it is relevant into our compilation 
> flags...
> 
> How did you managed to extract the libs from all the different flavors of MPI 
> (mpich, openmpi, intelmpi) so I can now reproduce what you did before?
> 
> (maybe this should have been my question... )

  Eric,

   Your problem is a result of Jed's constant bitching that we always list many 
libraries either redundantly or unnecessarily when we create our massive link 
line that allowed you this flexibility of linking with the "wrong" compiler. I 
changed the tests to not generate some redundant and unnecessary libraries to 
make Jed slightly less bitchy but missed the C++ one because no one uses it 
(except you). It would not be trivial for me to add the test because it would 
fail on any newer MPI that does not provide C++ bindings.

   Jed,

  Do all MPI's still provide the C++ bindings and will they forever so I 
can just directly put in an MPI C++ function call in the test? 

   Barry

> 
> Thanks again,
> 
> Eric
> 
> Le 18-02-10 à 12:37, Smith, Barry F. a écrit :
>>   Eric,
>> 
>>We need the entire link line.
>> 
>>What linker are you using C, or C++?  This is important.
>> 
>>Do you have dependencies on MPI C++ symbols? In other words, are you 
>> using C++ MPI bindings?
>> 
>> I cannot explain why all the libraries you listed would disappear from 
>> PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass 
>> under some circumstances less system (including MPI libraries) explicitly 
>> since they are already usually passed by the linker. This may be causing 
>> your difficulties.
>> 
>> Eagerly awaiting your reply.
>> 
>>  Barry
>> 
>> 
>>> On Feb 10, 2018, at 11:29 AM, Éric Chamberland 
>>>  wrote:
>>> 
>>> Hi Matthew,
>>> 
>>> Yes, I heard that the MPI C++ API has been deprecated.
>>> 
>>> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
>>> PETSC_WITH_EXTERNAL_LIB variable that is now:
>>> 
>>> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
>>> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
>>> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>>> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>>> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>>> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
>>> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc 
>>> -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps 
>>> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml 
>>> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd 
>>> -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread 
>>> -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis 
>>> -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm 
>>> -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm 
>>> -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ 
>>> -ldl
>>> We add some stuff to this (our own compiled libs), but nothing related to 
>>> MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our 
>>> diffrement environments (think about different petsc version, MPI libs and 
>>> compilers: clang, icc, g++) and it used to work until yesterday changes...
>>> 
>>> As you can see in the diff, this "block" of libraries have been removed 
>>> from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>>> --ldl
>>> --lmpi_cxx
>>> --lmpi
>>> --lstdc++
>>> --lm
>>> --lgcc_s
>>> --lpthread
>>> 
>>> and into this block there was the "-lmpi_cxx" that we need...
>>> 
>>> I could send you our whole line of link, but the error is into this small 
>>> change introduced yesterday into master...
>>> 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

  Eric,

How difficult would it be for you guys to switch to doing things 
"correctly" and not mixing up C++ compilers/linkers?

  Barry


> On Feb 10, 2018, at 11:51 AM, Éric Chamberland 
>  wrote:
> 
> Ok guys!
> 
> You found it (again...).  We are no using mpic++ to link!!!
> 
> We are not always using the same compiler that we used to compile petsc+mpi, 
> so we do not use mpic++ to compile (excepted for intelmpi).
> 
> For example, I compile MPI+Petsc with g++, but I am working with clang++ and 
> may switch to g++ for my day to day work.  This is perfectly working since... 
> long long time ago (I won't reveal my age... ;) )
> 
> I understand you choice, but your PETSC_WITH_EXTERNAL_LIB variable was 
> helping me to painlessly find the good linking options.
> 
> I can work to manually add it when it is relevant into our compilation 
> flags...
> 
> How did you managed to extract the libs from all the different flavors of MPI 
> (mpich, openmpi, intelmpi) so I can now reproduce what you did before?
> 
> (maybe this should have been my question... )
> 
> Thanks again,
> 
> Eric
> 
> Le 18-02-10 à 12:37, Smith, Barry F. a écrit :
>>   Eric,
>> 
>>We need the entire link line.
>> 
>>What linker are you using C, or C++?  This is important.
>> 
>>Do you have dependencies on MPI C++ symbols? In other words, are you 
>> using C++ MPI bindings?
>> 
>> I cannot explain why all the libraries you listed would disappear from 
>> PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass 
>> under some circumstances less system (including MPI libraries) explicitly 
>> since they are already usually passed by the linker. This may be causing 
>> your difficulties.
>> 
>> Eagerly awaiting your reply.
>> 
>>  Barry
>> 
>> 
>>> On Feb 10, 2018, at 11:29 AM, Éric Chamberland 
>>>  wrote:
>>> 
>>> Hi Matthew,
>>> 
>>> Yes, I heard that the MPI C++ API has been deprecated.
>>> 
>>> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with 
>>> PETSC_WITH_EXTERNAL_LIB variable that is now:
>>> 
>>> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib 
>>> -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib 
>>> -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 
>>> -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib 
>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>>> -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>>> -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 
>>> -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 
>>> -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc 
>>> -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps 
>>> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml 
>>> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd 
>>> -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread 
>>> -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis 
>>> -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm 
>>> -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm 
>>> -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ 
>>> -ldl
>>> We add some stuff to this (our own compiled libs), but nothing related to 
>>> MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our 
>>> diffrement environments (think about different petsc version, MPI libs and 
>>> compilers: clang, icc, g++) and it used to work until yesterday changes...
>>> 
>>> As you can see in the diff, this "block" of libraries have been removed 
>>> from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>>> --ldl
>>> --lmpi_cxx
>>> --lmpi
>>> --lstdc++
>>> --lm
>>> --lgcc_s
>>> --lpthread
>>> 
>>> and into this block there was the "-lmpi_cxx" that we need...
>>> 
>>> I could send you our whole line of link, but the error is into this small 
>>> change introduced yesterday into master...
>>> 
>>> Thanks a lot!
>>> 
>>> Eric
>>> 
>>> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
 On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland 
  wrote:
 Hi,
 
 we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB 
 variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and 
 everything was fine until this night.
 
 It seems some libs have vanished from this variable, see the diff here:
 
  -lptscotcherr
  -lesmumps
  -lscotch
  -lscotcherr
  -lm
  -lX11
 +-lstdc++
  -ldl
  -lmpi_usempi
  -lmpi_mpifh
  -lmpi
  -lgfortran

Re: [petsc-dev] MPI_Attr_get test fails

[Mark Adams]

On Sat, Feb 10, 2018 at 12:54 PM, Jed Brown  wrote:

> Mark Adams  writes:
>
> > On Fri, Feb 9, 2018 at 9:39 PM, Jeff Hammond 
> wrote:
> >
> >> https://msdn.microsoft.com/en-us/library/dn473234(v=vs.85).aspx
> >>
> >> This function name is archaic and removed in MPI-3. Use the new name.
> >>
> >>
> > OK, Jed did this a few months ago.
> >
> > It still fails because my code uses PETSC_COMM_WORLD. I guess I need to
> > change that.
>
> This runs cleanly with MPICH and Open MPI.  Mark, PETSC_COMM_WORLD is
> _usually_ equivalent to MPI_COMM_WORLD.  You should *never* use it in
> PETSc library code.


Now you tell me! 23 years later.


> It is not a "PETSc communicator" or "inner
> communicator" -- those terms refer to a dup'd communicator held by PETSc
> objects for PETSc's exclusive use.
>
>

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Le 18-02-10 à 12:38, Jed Brown a écrit :

Éric Chamberland  writes:



and into this block there was the "-lmpi_cxx" that we need...

The point is that if you are linking C++ code that calls the MPI C++
interface, then *you* should link with mpicxx or equivalent.
The funny thing, is that we are hopefully *not* using the C++ API of 
MPI.  We do use the C API since MPI 1.0.
Then, I am asking myself why this link error shows up now...since 
nothing calls it... hmmm, let me dig into this...

You should not depend on PETSc to provide anything but PETSc to your
application (so if you call other libraries that your configuration of
PETSc uses, you should take responsibility to link them explicitly; this
Wait: you mean for example if I configure MUMPS (or not) with PETSc, I 
have to build my link line with all MUMPS dependencies or not myself?
I wanted to be "lazy" and to use the same line *you* are using for 
passing libs (and in the good *order* which is not easy to manage)... as 
I can rely on this line to link "pure" petsc examples... ;)

is important when using shared libraries).  But you should definitely
not depend on PETSc to provide your application with stuff that has been
REMOVED from MPI (more than five years ago) and that PETSc does not use.
I agree, this C++ MPI API is not a very good excuse to add boring work 
into any software configuration and maintenance.


Thanks,

Eric

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Éric Chamberland]

Le 18-02-10 à 12:58, Smith, Barry F. a écrit :



On Feb 10, 2018, at 11:51 AM, Éric Chamberland 
 wrote:

Ok guys!

You found it (again...).  We are no using mpic++ to link!!!

We are not always using the same compiler that we used to compile petsc+mpi, so 
we do not use mpic++ to compile (excepted for intelmpi).

For example, I compile MPI+Petsc with g++, but I am working with clang++ and 
may switch to g++ for my day to day work.  This is perfectly working since... 
long long time ago (I won't reveal my age... ;) )

WOW. You are lucky this worked for you. I don't think missing C++ 
compilers/linkers is ever safe?
Honestly, I think yes it is safe under some constraints (not too 
different compilers).


To be sure of it we are constantly verifying our code under more than 30 
different mixes of compiler/petsc/mpi/Os each night since 17 years... 
mixing (or not) compiler used for compiling petsc vs our code with 100% 
reproducible results of our more than 2400 different validation tests 
(finite element code).





   Eric,

Your problem is a result of Jed's constant bitching that we always list many 
libraries either redundantly or unnecessarily when we create our massive link line that 
allowed you this flexibility of linking with the "wrong" compiler. I changed 
the tests to not generate some redundant and unnecessary libraries to make Jed slightly 
less bitchy but missed the C++ one because no one uses it (except you). It would not be 
trivial for me to add the test because it would fail on any newer MPI that does not 
provide C++ bindings.

Ok, but have you tested with openmpi 1.10.x?

Maybe this is the offending library, since we do not use the MPI C++ 
API...???


Maybe I can just dump support for this "old" mpi version and it will be 
fine...


Eric

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

> Le 18-02-10 à 12:38, Jed Brown a écrit :
>> Éric Chamberland  writes:
>>
>>>
>>> and into this block there was the "-lmpi_cxx" that we need...
>> The point is that if you are linking C++ code that calls the MPI C++
>> interface, then *you* should link with mpicxx or equivalent.
> The funny thing, is that we are hopefully *not* using the C++ API of 
> MPI.  We do use the C API since MPI 1.0.
> Then, I am asking myself why this link error shows up now...since 
> nothing calls it... hmmm, let me dig into this...

Is StatistiqueMemoire not your code?

| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Op::Init(void (*)(void const*, void*, int, 
| MPI::Datatype const&), bool)':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
reference to `ompi_mpi_cxx_op_intercept'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Intracomm::Create_graph(int, int const*, int const*, 
| bool) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Intercomm::Merge(bool) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Intracomm::Split(int, int) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Intracomm::Create(MPI::Group const&) const':
| /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
undefined reference to `MPI::Comm::Comm()'
| 
/pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
 
| In function `MPI::Intracomm::Clone() const':

>> You should not depend on PETSc to provide anything but PETSc to your
>> application (so if you call other libraries that your configuration of
>> PETSc uses, you should take responsibility to link them explicitly; this
> Wait: you mean for example if I configure MUMPS (or not) with PETSc, I 
> have to build my link line with all MUMPS dependencies or not myself?

Do you call MUMPS directly or through the PETSc interface?

Think about this from the perspective of packaging with shared
libraries, where we ask what needs to be updated when interfaces change.
Suppose we have these interface dependencies:

  libpetsc : libmumps libmpi
  app : libpetsc libmpi

Since MUMPS makes no guarantee of binary compatibility between releases,
updating MUMPS 5.1.2 to 5.1.3 requires rebuilding everything that link
to it.  Since PETSc calls MUMPS directly, libpetsc must be rebuilt.  If
your App does not call MUMPS directly, then you should link it with

  -lpetsc -lmpi

Now your binaries continue to work after libpetsc has been updated to
link the newer libmumps.  If you needlessly linked libmumps without
calling it directly, then you would also need to rebuild your App.  That
is called overlinking and is prohibited by most packaging guidelines
because it wastes lots of time and bandwidth for maintainers and users.

If you use pkg-config with PETSc, you get only -lpetsc when linking with
shared libraries.  (Pkg-config will give you everything for static
linking because it is needed and the concept of overlinking doesn't
exist in the same sense, though removing excess from the link line is
still desirable to make it easier to read and faster to link.)

Note that PETSc's own dependency management is messy because someone in
the early days of BuildSystem thought that concatenating strings was
sufficient, instead of maintaining a structured dependency graph to be
topologically sorted at the final stage (pkg-config does this).

> I wanted to be "lazy" and to use the same line *you* are using for 
> passing libs (and in the good *order* which is not easy to manage)... as 
> I can rely on this line to link "pure" petsc examples... ;)
>> is important when using shared libraries).  But you should definitely
>> not depend on PETSc to provide your application with stuff that has been
>> REMOVED from MPI (more than five years ago) and that PETSc does not use.
> I agree, this C++ MPI API is not a very good excuse to add boring work 
> into any software configuration and maintenance.

Re: [petsc-dev] MPI_Attr_get test fails

[Mark Adams]

On Fri, Feb 9, 2018 at 9:39 PM, Jeff Hammond  wrote:

> https://msdn.microsoft.com/en-us/library/dn473234(v=vs.85).aspx
>
> This function name is archaic and removed in MPI-3. Use the new name.
>
>
OK, Jed did this a few months ago.

It still fails because my code uses PETSC_COMM_WORLD. I guess I need to
change that.

Thanks,



> Jeff
>
> On Fri, Feb 9, 2018 at 3:11 PM Mark Adams  wrote:
>
>> I get an error in PetscCommGetNewTag. So I copied the test to may main
>> and I get the same problem. So this code fails:
>>
>> int main( int argc, char **args )
>> {
>>   PetscErrorCode   ierr;
>>   ierr = PetscInitialize( , , "./.petscrc", NULL
>> );CHKERRQ(ierr);
>>   { // debug
>> PetscCommCounter *counter;
>> PetscMPIInt  flg;
>> ierr = MPI_Attr_get(PETSC_COMM_WORLD,Petsc_Counter_keyval,,
>> );CHKERRQ(ierr);
>> if (!flg) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_CORRUPT,"Bad MPI
>> communicator supplied ???");
>>   }
>>
>> Any ideas?
>>
>> Mark
>>
> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
>

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Matthew Knepley]

On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland <
eric.chamberl...@giref.ulaval.ca> wrote:

> Hi,
>
> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB
> variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and
> everything was fine until this night.
>
> It seems some libs have vanished from this variable, see the diff here:
>
>  -lptscotcherr
>  -lesmumps
>  -lscotch
>  -lscotcherr
>  -lm
>  -lX11
> +-lstdc++
>  -ldl
>  -lmpi_usempi
>  -lmpi_mpifh
>  -lmpi
>  -lgfortran
>  -lm
>  -lgfortran
>  -lm
>  -lgcc_s
>  -lquadmath
>  -lpthread
> --ldl
> --lmpi_cxx
> --lmpi
> --lstdc++
> --lm
> --lgcc_s
> --lpthread
>  -lrt
>  -lm
>  -lpthread
>  -lz
> +-lstdc++
>  -ldl
>
>
> causing these errors at link phase for us:
>  
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype 
> const&), bool)':
> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122: undefined 
> reference to `ompi_mpi_cxx_op_intercept' 
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Intracomm::Create_graph(int, int const*, int const*, bool) 
> const':
> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25: undefined 
> reference to `MPI::Comm::Comm()' 
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Intercomm::Merge(bool) const':
> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
> undefined reference to `MPI::Comm::Comm()' 
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Intracomm::Split(int, int) const':
> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
> undefined reference to `MPI::Comm::Comm()' 
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Intracomm::Create(MPI::Group const&) const':
> /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23: 
> undefined reference to `MPI::Comm::Comm()' 
> /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
>  In function `MPI::Intracomm::Clone() const':
>
> Hi Eric,

These symbols are all coming from -lmpi_cxx. I would note that I believe
the MPI Forum has deprecated the C++ interface, so it will
eventually go away. However, lets fix this. In the configure log, I see
that mpicxx -show has that library in it, so it seems that the link
is not being done with the C++ compiler. Can you send the whole link line?

  Thanks,

 Matt


> Is this a normal and definitive change or an unwanted/unobserved bug?
>
> Thanks,
>
> Eric
>
> ps: here are the logs:
>
> this night:
>
> -
>
> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.loghttp://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log
>
>
> a day before:
> 
> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.loghttp://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Éric Chamberland  writes:

>> WOW. You are lucky this worked for you. I don't think missing C++ 
>> compilers/linkers is ever safe?
> Honestly, I think yes it is safe under some constraints (not too 
> different compilers).

C++ compilers on a given platform usually specify binary compatibility.
That is certainly the case for g++, clang++, and Intel icpc on
Linux/Mac/BSD, and also for MSVC and Intel on Windows.  Fortran
compilers generally don't do this.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

> On Feb 10, 2018, at 12:23 PM, Éric Chamberland 
>  wrote:
> 
> 
> Ok, but have you tested with openmpi 1.10.x?
> 
> Maybe this is the offending library, since we do not use the MPI C++ API...???

   Now I don't understand. Do you use the MPI C++ bindings or not? If you do 
not use the bindings then I don't see why they need to be resolved.

   Or is it because you link with the C++ compiler directly and not the MPI 
compiler wrapper?

  Barry

> 
> Maybe I can just dump support for this "old" mpi version and it will be 
> fine...
> 
> Eric
> 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Matthew Knepley]

On Sat, Feb 10, 2018 at 1:44 PM, Jed Brown  wrote:

> Éric Chamberland  writes:
>
> > Le 18-02-10 à 12:38, Jed Brown a écrit :
> >> Éric Chamberland  writes:
> >>
> >>>
> >>> and into this block there was the "-lmpi_cxx" that we need...
> >> The point is that if you are linking C++ code that calls the MPI C++
> >> interface, then *you* should link with mpicxx or equivalent.
> > The funny thing, is that we are hopefully *not* using the C++ API of
> > MPI.  We do use the C API since MPI 1.0.
> > Then, I am asking myself why this link error shows up now...since
> > nothing calls it... hmmm, let me dig into this...
>
> Is StatistiqueMemoire not your code?
>
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Op::Init(void (*)(void const*, void*, int,
> | MPI::Datatype const&), bool)':
> | /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/op_inln.h:122:
> undefined reference to `ompi_mpi_cxx_op_intercept'
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Intracomm::Create_graph(int, int const*, int const*,
> | bool) const':
> | /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> undefined reference to `MPI::Comm::Comm()'
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Intercomm::Merge(bool) const':
> | /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:
> undefined reference to `MPI::Comm::Comm()'
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Intracomm::Split(int, int) const':
> | /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:
> undefined reference to `MPI::Comm::Comm()'
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Intracomm::Create(MPI::Group const&) const':
> | /opt/openmpi-1.10.2/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:
> undefined reference to `MPI::Comm::Comm()'
> | /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/
> COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o:
> | In function `MPI::Intracomm::Clone() const':
>
> >> You should not depend on PETSc to provide anything but PETSc to your
> >> application (so if you call other libraries that your configuration of
> >> PETSc uses, you should take responsibility to link them explicitly; this
> > Wait: you mean for example if I configure MUMPS (or not) with PETSc, I
> > have to build my link line with all MUMPS dependencies or not myself?
>
> Do you call MUMPS directly or through the PETSc interface?
>
> Think about this from the perspective of packaging with shared
> libraries, where we ask what needs to be updated when interfaces change.
> Suppose we have these interface dependencies:
>
>   libpetsc : libmumps libmpi
>   app : libpetsc libmpi
>
> Since MUMPS makes no guarantee of binary compatibility between releases,
> updating MUMPS 5.1.2 to 5.1.3 requires rebuilding everything that link
> to it.  Since PETSc calls MUMPS directly, libpetsc must be rebuilt.  If
> your App does not call MUMPS directly, then you should link it with
>
>   -lpetsc -lmpi
>
> Now your binaries continue to work after libpetsc has been updated to
> link the newer libmumps.  If you needlessly linked libmumps without
> calling it directly, then you would also need to rebuild your App.  That
> is called overlinking and is prohibited by most packaging guidelines
> because it wastes lots of time and bandwidth for maintainers and users.
>
> If you use pkg-config with PETSc, you get only -lpetsc when linking with
> shared libraries.  (Pkg-config will give you everything for static
> linking because it is needed and the concept of overlinking doesn't
> exist in the same sense, though removing excess from the link line is
> still desirable to make it easier to read and faster to link.)
>
> Note that PETSc's own dependency management is messy because someone in
> the early days of BuildSystem thought that concatenating strings was
> sufficient, instead of maintaining a structured dependency graph to be
> topologically sorted at the final stage (pkg-config does this).
>

Of course, we do have a structured dependency graph. Where is it not being
used?

   Matt


> > I wanted to be "lazy" and to use the same line *you* are using for
> > passing libs (and in the good *order* which is not easy to manage)... as
> > I can rely on this line to link "pure" petsc examples... ;)
> >> is important when using shared libraries).  But you should definitely
> >> not depend on PETSc to provide your application with stuff that has been
> >> REMOVED from MPI (more than five years ago) and that PETSc does not use.
> > I 

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

"Smith, Barry F."  writes:

>   Do all MPI's still provide the C++ bindings and will they
>   forever so I can just directly put in an MPI C++ function call
>   in the test?

I would not count on that.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Smith, Barry F.]

> On Feb 10, 2018, at 1:18 PM, Jed Brown  wrote:
> 
> "Smith, Barry F."  writes:
> 
>>  Do all MPI's still provide the C++ bindings and will they
>>  forever so I can just directly put in an MPI C++ function call
>>  in the test?
> 
> I would not count on that.

   Then the testing is more involved, need to first check if C++ bindings exist 
and then check if supplied automatically by compilers (like on Cray). Annoying, 
I'd prefer to avoid but I still don't have the definitive word if C++ bindings 
are the issue from Eric.

  

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

Matthew Knepley  writes:

>> Note that PETSc's own dependency management is messy because someone in
>> the early days of BuildSystem thought that concatenating strings was
>> sufficient, instead of maintaining a structured dependency graph to be
>> topologically sorted at the final stage (pkg-config does this).
>>
>
> Of course, we do have a structured dependency graph. Where is it not being
> used?

If the DAG structure was preserved, toStringNoDupes would never need to
work with a list containing duplicate flags, including libraries like
-ldl and -lm, because the topological sort would never have created
duplicates in the first place.

Re: [petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

[Jed Brown]

"Smith, Barry F."  writes:

>Then the testing is more involved, need to first check if C++ bindings 
> exist and then check if supplied automatically by compilers (like on Cray). 
> Annoying, I'd prefer to avoid but I still don't have the definitive word if 
> C++ bindings are the issue from Eric.

The undefined symbols are C++ bindings and they are referenced from a
source file named StatistiqueMemoire.* which is not part of PETSc or a
PETSc dependency.  We don't reference them so we shouldn't try to link
them.