Hi all,
A discussion came up recently around including rpaths in our mesos
binaries to help resolve any shared library dependencies that don't
exist in standard library paths (e.g. /lib, /usr/local/lib, etc.).
By default, there are no shared library dependencies that exist in
non-standard paths, because we bundle all of these dependencies into
the mesos source and statically link them into our executables (e.g.
glog, zookeeper, etc.)
However, if you configure mesos with e.g.
../configure ----disable-bundled
or the more selective
../configure --with-glog[=DIR] --with-zookeeper[=DIR] ...
then mesos will be built with an external shared library dependency
(e.g. glog and zookeeper in this case).
The build system is smart enough to set up LDFLAGS so we can link
against whatever external libraries are passed in via the --with-*
flags.
However, when we go to run the binaries that are produced (e.g.
mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them
with an LD_LIBRARY_PATH pointing to the location of the shared
libraires from these external dependencies, e.g.
LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master
It would be nice if we didn't have to explicitly set the
LD_LIBRARY_PATH to launch these binaries when linking against any
external shared library dependencies.
One way around this would be to make sure that all external library
dependencies were stored in standard search paths for the dynamic
linker. This is typically what happens if you install these
dependencies via a standard package manager (e.g. apt-get, yum, etc.).
Sometimes this is undesirable (or impossible) though, especially if
the external dependencies do not exist as packages or follow a
non-standard directory hierarchy in terms of where it places its
include files, libraries, etc.
Another option is to install the paths to these external libraries
into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the
dynamic linker will search them at runtime. This is also unfeasible
at times and has the added disadvantage that these library paths will
now be searched for *all* binaries that get executed (not just the
ones we currently care about).
The final option (and the one I'm proposing here) is to set the
'rpath' of the binary to point to the location of the external shared
library on the build machine. The rpath is embedded into the binary
at link time and is used to give the linker an extra set of paths to
search for shared libraries at runtime. This obvious advantage here
is that setting rpath allows us to run our binaries without requiring
LD_LIBRARY_PATH or any of the other methods mentioned above to tell
the linker where to find our shared libraries. However, it has the
disadvantage of baking a path into the binary that may only exist on
the specific machine the binary was built on.
That said, the standard search order used by the dynamic linker to
find shared libraries is:
1) LD_LIBRARY_PATH
2) rpath
4) the ldcache (/etc/ld.so.conf on linux)
3) default paths (e.g. /lib, /usr/local/lib)
Meaning that we could always overwrite the rpath using LD_LIBARY_PATH
if we wanted to. Moreover, we could even change the rpath at the time
of deployment (e.g. via chrpath on linux). This may be desirable if
the shared libraries are installed at different locations on the
deployment machine.
If there are no objections, I therefore propose we modify the
following files to add rpaths to all external dependencies set via
--with-* flags:
./configure.ac
./3rdparty/libprocess/3rdparty/stout/configure.ac
./3rdparty/libprocess/configure.ac
The pattern would change from:
CPPFLAGS="-I${with_thing}/include $CPPFLAGS"
LDFLAGS="-L${with_thing}/lib $LDFLAGS"
to include an additional line with:
LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS"
I know there has some hesitation with this in the past (especially
when it comes to producing rpms or debs, where baking in an rpath
seems really strange), but I'm hoping people will agree that it's
useful enough that it makes sense to include the rpaths as the default
case. We can always run a post-processing step to strip them in cases
where they are undesirable.
Thanks!
--
~Kevin
