Playing with this a bit more, I came up with a solution that bridges
the gap between the two alternatives I proposed before. Basically, I
found a way to get the wrapper script to set the rpath when running
the binary from the build directory, but leave the rpath of the actual
binary unchanged. Best of both worlds.
Instead of explicity setting the global LDFLAGS in configure.ac, I now
set an intermediate variable called LIBMESOS_RPATH for all non-bundled
packages:
LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_zookeeper}/lib"
LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_glog}/lib"
...
Then at the bottom of configure.ac, I do:
AC_SUBST(LIBMESOS_RPATH, ["${LIBMESOS_RPATH//:/-rpath }"])
This substitutes all colons in LIBMESOS_RPATH with "-rpath " to form a
valid string of rpaths for consumption by Makefile.am. I chose to
format the original LIBMESOS_RPATH string with colons so that it forms
a valid LD_LIBRARY_PATH as well (plus it's easier to tokenize by ':'
than by '-rpath ').
Then in Makefile.am I do:
libmesos_la_LDFLAGS += $(LIBMESOS_RPATH)
This sets the rpath for libmesos using *libtool's* LDFLAGS rather than
setting the global LDFLAGS for all linked objects (as I was doing
before).
With this change, I can inspect libtools wrapper scripts to see that indeed
it patches up the binary with the added rpaths, but libmesos itself
has no rpath set.
Pending any objections, a RR will be forthcoming.
On Fri, Feb 12, 2016 at 3:31 PM, Kevin Klues <klue...@gmail.com> wrote:
> I like that idea. Don't set rpath by default, but allow people to
> specify that it should be set via a flag. How does
>
> --set-rpath-for-external-libs
>
> sound for a name. Too long? I don't like just --with-rpath, because
> it's not descriptive enough in my opinion.
>
> On Fri, Feb 12, 2016 at 2:34 PM, Jojy Varghese <j...@mesosphere.io> wrote:
>> Maybe have an opt-in (say —with-rpath)?
>>
>> -Jojy
>>
>>> On Feb 12, 2016, at 2:19 PM, Kevin Klues <klue...@gmail.com> wrote:
>>>
>>> To be clear, I'm actually a bit torn both ways on this.
>>>
>>> On the one hand, including the rpath makes it easy for those who don't
>>> know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass
>>> their paths to their external dependencies at configure time and then
>>> run their binaries without further effort.
>>>
>>> On the other hand, maybe they should be cognizant of the fact that
>>> something is going on under the hood to actually allow their binaries
>>> to link properly (i.e. I can imagine a situation where someone builds,
>>> runs, and tests everything locally, and then is confused as to why it
>>> nothing works once deployed).
>>>
>>> In my previous email, I argue that we should include the rpath by
>>> default, and can strip it later if we don't want it for some reason
>>> (i.e. when bundling into debs/rpms). Conversely, we could leave it
>>> out by default and only set it as a post-processing step in situations
>>> where we actually care about it.
>>>
>>> I'm curious what other people's thoughts are.
>>>
>>>
>>> On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <klue...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> A discussion came up recently around including rpaths in our mesos
>>>> binaries to help resolve any shared library dependencies that don't
>>>> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.).
>>>>
>>>> By default, there are no shared library dependencies that exist in
>>>> non-standard paths, because we bundle all of these dependencies into
>>>> the mesos source and statically link them into our executables (e.g.
>>>> glog, zookeeper, etc.)
>>>>
>>>> However, if you configure mesos with e.g.
>>>>
>>>> ../configure ----disable-bundled
>>>>
>>>> or the more selective
>>>>
>>>> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ...
>>>>
>>>> then mesos will be built with an external shared library dependency
>>>> (e.g. glog and zookeeper in this case).
>>>>
>>>> The build system is smart enough to set up LDFLAGS so we can link
>>>> against whatever external libraries are passed in via the --with-*
>>>> flags.
>>>>
>>>> However, when we go to run the binaries that are produced (e.g.
>>>> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them
>>>> with an LD_LIBRARY_PATH pointing to the location of the shared
>>>> libraires from these external dependencies, e.g.
>>>>
>>>> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master
>>>>
>>>> It would be nice if we didn't have to explicitly set the
>>>> LD_LIBRARY_PATH to launch these binaries when linking against any
>>>> external shared library dependencies.
>>>>
>>>> One way around this would be to make sure that all external library
>>>> dependencies were stored in standard search paths for the dynamic
>>>> linker. This is typically what happens if you install these
>>>> dependencies via a standard package manager (e.g. apt-get, yum, etc.).
>>>> Sometimes this is undesirable (or impossible) though, especially if
>>>> the external dependencies do not exist as packages or follow a
>>>> non-standard directory hierarchy in terms of where it places its
>>>> include files, libraries, etc.
>>>>
>>>> Another option is to install the paths to these external libraries
>>>> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the
>>>> dynamic linker will search them at runtime. This is also unfeasible
>>>> at times and has the added disadvantage that these library paths will
>>>> now be searched for *all* binaries that get executed (not just the
>>>> ones we currently care about).
>>>>
>>>> The final option (and the one I'm proposing here) is to set the
>>>> 'rpath' of the binary to point to the location of the external shared
>>>> library on the build machine. The rpath is embedded into the binary
>>>> at link time and is used to give the linker an extra set of paths to
>>>> search for shared libraries at runtime. This obvious advantage here
>>>> is that setting rpath allows us to run our binaries without requiring
>>>> LD_LIBRARY_PATH or any of the other methods mentioned above to tell
>>>> the linker where to find our shared libraries. However, it has the
>>>> disadvantage of baking a path into the binary that may only exist on
>>>> the specific machine the binary was built on.
>>>>
>>>> That said, the standard search order used by the dynamic linker to
>>>> find shared libraries is:
>>>>
>>>> 1) LD_LIBRARY_PATH
>>>> 2) rpath
>>>> 4) the ldcache (/etc/ld.so.conf on linux)
>>>> 3) default paths (e.g. /lib, /usr/local/lib)
>>>>
>>>> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH
>>>> if we wanted to. Moreover, we could even change the rpath at the time
>>>> of deployment (e.g. via chrpath on linux). This may be desirable if
>>>> the shared libraries are installed at different locations on the
>>>> deployment machine.
>>>>
>>>> If there are no objections, I therefore propose we modify the
>>>> following files to add rpaths to all external dependencies set via
>>>> --with-* flags:
>>>>
>>>> ./configure.ac
>>>> ./3rdparty/libprocess/3rdparty/stout/configure.ac
>>>> ./3rdparty/libprocess/configure.ac
>>>>
>>>> The pattern would change from:
>>>>
>>>> CPPFLAGS="-I${with_thing}/include $CPPFLAGS"
>>>> LDFLAGS="-L${with_thing}/lib $LDFLAGS"
>>>>
>>>> to include an additional line with:
>>>>
>>>> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS"
>>>>
>>>> I know there has some hesitation with this in the past (especially
>>>> when it comes to producing rpms or debs, where baking in an rpath
>>>> seems really strange), but I'm hoping people will agree that it's
>>>> useful enough that it makes sense to include the rpaths as the default
>>>> case. We can always run a post-processing step to strip them in cases
>>>> where they are undesirable.
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> ~Kevin
>>>
>>>
>>>
>>> --
>>> ~Kevin
>>
>
>
>
> --
> ~Kevin
--
~Kevin
