Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jed Brown]

On Thu, May 31, 2012 at 6:20 AM, Jeff Squyres  wrote:

> On May 29, 2012, at 11:42 AM, Jed Brown wrote:
>
> > The pkg-config approach is to use pkg-config --static if you want to
> link that library statically.
>
> Do the OMPI pkg-config files not do this properly?
>

Looks right to me. I think the complaint was that there was no way to
specify the equivalent using wrapper compilers. I don't like the wrapper
compiler model (certainly not for languages with a common ABI like C), but
pkg-config doesn't have a good way to manage multiple configurations.


>
> > So the problem is almost exclusively one of binary compatibility. If an
> app or library is only linked to the interface libs, underlying system
> libraries can be upgraded to different soname without needing to relink the
> applications. For example, libhwloc could be upgraded to a different ABI,
> Open MPI rebuilt, and then the user application and intermediate MPI-based
> libraries would not need to be rebuilt. This is great for distributions and
> convenient if you work on projects with lots of dependencies.
> >
> > It's not such an issue for HPC applications because we tend to recompile
> a lot and don't need binary compatibility for many of the most common use
> cases. There is also the linker option -Wl,--as-needed that usually does
> what is desired.
>
> Mmmm.  Ok.  Brian and I are going to be in the same physical location next
> week; I'll chat with him about this.

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jeff Squyres]

On May 29, 2012, at 11:42 AM, Jed Brown wrote:

> The pkg-config approach is to use pkg-config --static if you want to link 
> that library statically.

Do the OMPI pkg-config files not do this properly?

> So the problem is almost exclusively one of binary compatibility. If an app 
> or library is only linked to the interface libs, underlying system libraries 
> can be upgraded to different soname without needing to relink the 
> applications. For example, libhwloc could be upgraded to a different ABI, 
> Open MPI rebuilt, and then the user application and intermediate MPI-based 
> libraries would not need to be rebuilt. This is great for distributions and 
> convenient if you work on projects with lots of dependencies.
> 
> It's not such an issue for HPC applications because we tend to recompile a 
> lot and don't need binary compatibility for many of the most common use 
> cases. There is also the linker option -Wl,--as-needed that usually does what 
> is desired.

Mmmm.  Ok.  Brian and I are going to be in the same physical location next 
week; I'll chat with him about this.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jed Brown]

On Tue, May 29, 2012 at 9:05 AM, Jeff Squyres  wrote:
>
>
> We've tossed around ideas such as having the wrappers always assume
> dynamic linking (e.g., only include a minimum of libraries), and then add
> another wrapper option like --wrapper:static (or whatever) to know when to
> add in all the dependent libraries.  Or possibly even look for some popular
> linker options like --static, or some such (which we've tried to avoid,
> because that can turn into a slippery slope), but such switches aren't
> always necessary for MPI-only-static (vs. completely-100%-static) linking.
>  It gets even fuzzier when both libmpi.so and libmpi.a are present.  Which
> way should we assume?
>
> Another problem is backwards compatibility -- users who are currently
> statically linking will assume the old behavior (of not needing to specify
> anything additional).
>
> > Now I'm not saying that Open MPI should commit to pkg-config instead of
> wrapper compilers, but the concept of linking differently for static versus
> shared libraries is something that should be observed.
>
> Fair enough.  But we've never been able to come up with a rational way to
> do it (note that pkg-config has its own problems -- OMPI provides
> pkg-config files in addition to wrapper compilers, but they don't fix
> everything, either).
>
> We have users who both --enable-static and --enable-shared (meaning: both
> libmpi.so and libmpi.a are present).  And therefore we've come down on the
> conservative side of adding in whatever is necessary for static linking.
>

The pkg-config approach is to use pkg-config --static if you want to link
that library statically.


>
> > (Over-linking is an ongoing problem with HPC-oriented packages. We are
> probably all guilty of it, but tools like pkg-config don't handle multiple
> configurations well and I don't know of a similar system that manages both
> static/shared and multi-configuration well.)
>
> I suppose, but it does depend on how you define "problem".  The linker
> will ignore any unused libraries -- so it's a problem like lint is a
> problem.  It's annoying, but it doesn't do any harm.
>
> ...or are there cases where it actually does something harmful?
>

So the problem is almost exclusively one of binary compatibility. If an app
or library is only linked to the interface libs, underlying system
libraries can be upgraded to different soname without needing to relink the
applications. For example, libhwloc could be upgraded to a different ABI,
Open MPI rebuilt, and then the user application and intermediate MPI-based
libraries would not need to be rebuilt. This is great for distributions and
convenient if you work on projects with lots of dependencies.

It's not such an issue for HPC applications because we tend to recompile a
lot and don't need binary compatibility for many of the most common use
cases. There is also the linker option -Wl,--as-needed that usually does
what is desired.

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jeff Squyres]

On May 27, 2012, at 1:52 PM, Jed Brown wrote:

> On Wed, May 23, 2012 at 8:29 AM, Barrett, Brian W  wrote:
> 
> > And therein lies the problem.  We have a number of users who build Open
> > MPI statically and even some who build both static and shared libraries in
> > the same build.  We've never been able to figure out a reasonable way to
> > guess if we need to add -lhwloc or -ldl, so we add them.  It's better to
> > list them and have some redundant dependencies (since you have to have the
> > library anyways) than to not list them and have odd link errors.
> 
> So pkg-config has the --static option for exactly this reason. Let's look at 
> Cairo as an example.
> 
> $ cat /usr/lib/pkgconfig/cairo.pc 
> prefix=/usr
> exec_prefix=${prefix}
> [snip]
> $ pkg-config cairo --libs
> -lcairo
> $ pkg-config cairo --libs --static
> -pthread -lcairo -lgobject-2.0 -lffi -lpixman-1 -lfontconfig -lexpat 
> -lfreetype -lbz2 -lpng15 -lz -lm -lxcb-shm -lxcb-render -lXrender -lglib-2.0 
> -lrt -lpcre -lX11 -lpthread -lxcb -lXau -lXdmcp

We've tossed around ideas such as having the wrappers always assume dynamic 
linking (e.g., only include a minimum of libraries), and then add another 
wrapper option like --wrapper:static (or whatever) to know when to add in all 
the dependent libraries.  Or possibly even look for some popular linker options 
like --static, or some such (which we've tried to avoid, because that can turn 
into a slippery slope), but such switches aren't always necessary for 
MPI-only-static (vs. completely-100%-static) linking.  It gets even fuzzier 
when both libmpi.so and libmpi.a are present.  Which way should we assume?

Another problem is backwards compatibility -- users who are currently 
statically linking will assume the old behavior (of not needing to specify 
anything additional).

> Now I'm not saying that Open MPI should commit to pkg-config instead of 
> wrapper compilers, but the concept of linking differently for static versus 
> shared libraries is something that should be observed.

Fair enough.  But we've never been able to come up with a rational way to do it 
(note that pkg-config has its own problems -- OMPI provides pkg-config files in 
addition to wrapper compilers, but they don't fix everything, either).

We have users who both --enable-static and --enable-shared (meaning: both 
libmpi.so and libmpi.a are present).  And therefore we've come down on the 
conservative side of adding in whatever is necessary for static linking.

> (Over-linking is an ongoing problem with HPC-oriented packages. We are 
> probably all guilty of it, but tools like pkg-config don't handle multiple 
> configurations well and I don't know of a similar system that manages both 
> static/shared and multi-configuration well.)

I suppose, but it does depend on how you define "problem".  The linker will 
ignore any unused libraries -- so it's a problem like lint is a problem.  It's 
annoying, but it doesn't do any harm.

...or are there cases where it actually does something harmful?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jed Brown]

On Wed, May 23, 2012 at 8:29 AM, Barrett, Brian W wrote:

> >I should add the caveat that they are need when linking statically, but
> >not when using shared libraries.
>
> And therein lies the problem.  We have a number of users who build Open
> MPI statically and even some who build both static and shared libraries in
> the same build.  We've never been able to figure out a reasonable way to
> guess if we need to add -lhwloc or -ldl, so we add them.  It's better to
> list them and have some redundant dependencies (since you have to have the
> library anyways) than to not list them and have odd link errors.


So pkg-config has the --static option for exactly this reason. Let's look
at Cairo as an example.

$ cat /usr/lib/pkgconfig/cairo.pc
prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: cairo
Description: Multi-platform 2D graphics library
Version: 1.12.2

Requires.private:   gobject-2.0 glib-2.0   pixman-1 >= 0.22.0
 fontconfig >= 2.2.95 freetype2 >= 9.7.3   libpng xcb-shm xcb >= 1.6
xcb-render >= 1.6 xrender >= 0.6 x11
Libs: -L${libdir} -lcairo
Libs.private:-lz -lz
Cflags: -I${includedir}/cairo

$ pkg-config cairo --libs
-lcairo
$ pkg-config cairo --libs --static
-pthread -lcairo -lgobject-2.0 -lffi -lpixman-1 -lfontconfig -lexpat
-lfreetype -lbz2 -lpng15 -lz -lm -lxcb-shm -lxcb-render -lXrender
-lglib-2.0 -lrt -lpcre -lX11 -lpthread -lxcb -lXau -lXdmcp
$ ldd /usr/lib/libcairo.so
linux-vdso.so.1 =>  (0x7fff741ff000)
libpthread.so.0 => /lib/libpthread.so.0 (0x7f135eac7000)
libpixman-1.so.0 => /usr/lib/libpixman-1.so.0 (0x7f135e83f000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1
(0x7f135e608000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x7f135e369000)
libpng15.so.15 => /usr/lib/libpng15.so.15 (0x7f135e13c000)
libxcb-shm.so.0 => /usr/lib/libxcb-shm.so.0 (0x7f135df39000)
libxcb-render.so.0 => /usr/lib/libxcb-render.so.0
(0x7f135dd3)
libxcb.so.1 => /usr/lib/libxcb.so.1 (0x7f135db12000)
libXrender.so.1 => /usr/lib/libXrender.so.1 (0x7f135d906000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x7f135d5cc000)
libz.so.1 => /usr/lib/libz.so.1 (0x7f135d3b6000)
librt.so.1 => /lib/librt.so.1 (0x7f135d1ad000)
libm.so.6 => /lib/libm.so.6 (0x7f135ceb8000)
libc.so.6 => /lib/libc.so.6 (0x7f135cb17000)
/lib/ld-linux-x86-64.so.2 (0x7f135f012000)
libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x7f135c906000)
libexpat.so.1 => /usr/lib/libexpat.so.1 (0x7f135c6dc000)
libXau.so.6 => /usr/lib/libXau.so.6 (0x7f135c4d8000)
libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x7f135c2d1000)
libdl.so.2 => /lib/libdl.so.2 (0x7f135c0cd000)

Now I'm not saying that Open MPI should commit to pkg-config instead of
wrapper compilers, but the concept of linking differently for static versus
shared libraries is something that should be observed.

(Over-linking is an ongoing problem with HPC-oriented packages. We are
probably all guilty of it, but tools like pkg-config don't handle multiple
configurations well and I don't know of a similar system that manages both
static/shared and multi-configuration well.)

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Jeff Squyres]

On May 24, 2012, at 6:07 PM, Orion Poplawski wrote:

>>> I should add the caveat that they are need when linking statically, but
>>> not when using shared libraries.
>> 
>> And therein lies the problem.  We have a number of users who build Open
>> MPI statically and even some who build both static and shared libraries in
>> the same build.  We've never been able to figure out a reasonable way to
>> guess if we need to add -lhwloc or -ldl, so we add them.  It's better to
>> list them and have some redundant dependencies (since you have to have the
>> library anyways) than to not list them and have odd link errors.
> 
> Could you not add them if --enable-static is not passed to configure?  That 
> would at least help with building the Fedora packages.


I have a dim recollection that we specifically chose not to do this (i.e., that 
there was some hidden "gotcha" with that approach), but I confess to not 
remembering why.

Brian -- do you remember?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Orion Poplawski]

On 05/23/2012 07:29 AM, Barrett, Brian W wrote:

On 5/22/12 10:36 PM, "Orion Poplawski"  wrote:


On 05/22/2012 10:34 PM, Orion Poplawski wrote:

On 05/21/2012 06:15 PM, Jeff Squyres wrote:

On May 15, 2012, at 10:37 AM, Orion Poplawski wrote:


$ mpicc -showme:link
-pthread -m64 -L/usr/lib64/openmpi/lib -lmpi -ldl -lhwloc

-ldl and -lhwloc should not be listed. The user should only link
against libraries that they are using directly, namely -lmpi, and
they should explicitly add -ldl and -lhwloc if their code directly
uses those libraries. There does not appear to be any references to
libdl or libhwloc symbols in the openmpi headers which is the other
place it could come in.


I just read this a few times, and I admit that I'm a little confused.

libmpi does use symbols from libdl; we use it to load OMPI's plugins.
So I'm not sure why you're saying we shouldn't -ldl in the wrapper
compiler...?

libhwloc might be a little questionable here -- I'll have to check to
see whether 1.6 uses hwloc only in a plugin or whether it's used in
the base library (I don't remember offhand).



But libmpi is already linked against libdl and libhwloc. The wrapper
libraries are added when linking user code. But unless a user's code
directly uses libdl or libhwloc they don't need to link to those
libraries.



I should add the caveat that they are need when linking statically, but
not when using shared libraries.


And therein lies the problem.  We have a number of users who build Open
MPI statically and even some who build both static and shared libraries in
the same build.  We've never been able to figure out a reasonable way to
guess if we need to add -lhwloc or -ldl, so we add them.  It's better to
list them and have some redundant dependencies (since you have to have the
library anyways) than to not list them and have odd link errors.

We're open to better suggestions, but keep in mind that they have to be
portable and not require the user to change their build systems (ie, we
can't just depend on libtool to do everything for us).

Brian


Could you not add them if --enable-static is not passed to configure?  That 
would at least help with building the Fedora packages.



--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder Office  FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301   http://www.nwra.com

Re: [OMPI users] [EXTERNAL] Re: mpicc link shouldn't add -ldl and -lhwloc

[Barrett, Brian W]

On 5/22/12 10:36 PM, "Orion Poplawski"  wrote:

>On 05/22/2012 10:34 PM, Orion Poplawski wrote:
>> On 05/21/2012 06:15 PM, Jeff Squyres wrote:
>>> On May 15, 2012, at 10:37 AM, Orion Poplawski wrote:
>>>
 $ mpicc -showme:link
 -pthread -m64 -L/usr/lib64/openmpi/lib -lmpi -ldl -lhwloc

 -ldl and -lhwloc should not be listed. The user should only link
 against libraries that they are using directly, namely -lmpi, and
 they should explicitly add -ldl and -lhwloc if their code directly
 uses those libraries. There does not appear to be any references to
 libdl or libhwloc symbols in the openmpi headers which is the other
 place it could come in.
>>>
>>> I just read this a few times, and I admit that I'm a little confused.
>>>
>>> libmpi does use symbols from libdl; we use it to load OMPI's plugins.
>>> So I'm not sure why you're saying we shouldn't -ldl in the wrapper
>>> compiler...?
>>>
>>> libhwloc might be a little questionable here -- I'll have to check to
>>> see whether 1.6 uses hwloc only in a plugin or whether it's used in
>>> the base library (I don't remember offhand).
>>>
>>
>> But libmpi is already linked against libdl and libhwloc. The wrapper
>> libraries are added when linking user code. But unless a user's code
>> directly uses libdl or libhwloc they don't need to link to those
>>libraries.
>>
>
>I should add the caveat that they are need when linking statically, but
>not when using shared libraries.

And therein lies the problem.  We have a number of users who build Open
MPI statically and even some who build both static and shared libraries in
the same build.  We've never been able to figure out a reasonable way to
guess if we need to add -lhwloc or -ldl, so we add them.  It's better to
list them and have some redundant dependencies (since you have to have the
library anyways) than to not list them and have odd link errors.

We're open to better suggestions, but keep in mind that they have to be
portable and not require the user to change their build systems (ie, we
can't just depend on libtool to do everything for us).

Brian

-- 
  Brian W. Barrett
  Dept. 1423: Scalable System Software
  Sandia National Laboratories