I was wrong. This unset the rpath in the binary, but left the rpath set in libmesos.so. I am going to leave this as an open issue for now, pending a better solution.
On Fri, Feb 12, 2016 at 5:03 PM, Kevin Klues <[email protected]> wrote: > Playing with this a bit more, I came up with a solution that bridges > the gap between the two alternatives I proposed before. Basically, I > found a way to get the wrapper script to set the rpath when running > the binary from the build directory, but leave the rpath of the actual > binary unchanged. Best of both worlds. > > Instead of explicity setting the global LDFLAGS in configure.ac, I now > set an intermediate variable called LIBMESOS_RPATH for all non-bundled > packages: > > LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_zookeeper}/lib" > LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_glog}/lib" > ... > > Then at the bottom of configure.ac, I do: > > AC_SUBST(LIBMESOS_RPATH, ["${LIBMESOS_RPATH//:/-rpath }"]) > > This substitutes all colons in LIBMESOS_RPATH with "-rpath " to form a > valid string of rpaths for consumption by Makefile.am. I chose to > format the original LIBMESOS_RPATH string with colons so that it forms > a valid LD_LIBRARY_PATH as well (plus it's easier to tokenize by ':' > than by '-rpath '). > > Then in Makefile.am I do: > > libmesos_la_LDFLAGS += $(LIBMESOS_RPATH) > > This sets the rpath for libmesos using *libtool's* LDFLAGS rather than > setting the global LDFLAGS for all linked objects (as I was doing > before). > > With this change, I can inspect libtools wrapper scripts to see that indeed > it patches up the binary with the added rpaths, but libmesos itself > has no rpath set. > > Pending any objections, a RR will be forthcoming. > > On Fri, Feb 12, 2016 at 3:31 PM, Kevin Klues <[email protected]> wrote: >> I like that idea. Don't set rpath by default, but allow people to >> specify that it should be set via a flag. How does >> >> --set-rpath-for-external-libs >> >> sound for a name. Too long? I don't like just --with-rpath, because >> it's not descriptive enough in my opinion. >> >> On Fri, Feb 12, 2016 at 2:34 PM, Jojy Varghese <[email protected]> wrote: >>> Maybe have an opt-in (say —with-rpath)? >>> >>> -Jojy >>> >>>> On Feb 12, 2016, at 2:19 PM, Kevin Klues <[email protected]> wrote: >>>> >>>> To be clear, I'm actually a bit torn both ways on this. >>>> >>>> On the one hand, including the rpath makes it easy for those who don't >>>> know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass >>>> their paths to their external dependencies at configure time and then >>>> run their binaries without further effort. >>>> >>>> On the other hand, maybe they should be cognizant of the fact that >>>> something is going on under the hood to actually allow their binaries >>>> to link properly (i.e. I can imagine a situation where someone builds, >>>> runs, and tests everything locally, and then is confused as to why it >>>> nothing works once deployed). >>>> >>>> In my previous email, I argue that we should include the rpath by >>>> default, and can strip it later if we don't want it for some reason >>>> (i.e. when bundling into debs/rpms). Conversely, we could leave it >>>> out by default and only set it as a post-processing step in situations >>>> where we actually care about it. >>>> >>>> I'm curious what other people's thoughts are. >>>> >>>> >>>> On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <[email protected]> wrote: >>>>> Hi all, >>>>> >>>>> A discussion came up recently around including rpaths in our mesos >>>>> binaries to help resolve any shared library dependencies that don't >>>>> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.). >>>>> >>>>> By default, there are no shared library dependencies that exist in >>>>> non-standard paths, because we bundle all of these dependencies into >>>>> the mesos source and statically link them into our executables (e.g. >>>>> glog, zookeeper, etc.) >>>>> >>>>> However, if you configure mesos with e.g. >>>>> >>>>> ../configure ----disable-bundled >>>>> >>>>> or the more selective >>>>> >>>>> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ... >>>>> >>>>> then mesos will be built with an external shared library dependency >>>>> (e.g. glog and zookeeper in this case). >>>>> >>>>> The build system is smart enough to set up LDFLAGS so we can link >>>>> against whatever external libraries are passed in via the --with-* >>>>> flags. >>>>> >>>>> However, when we go to run the binaries that are produced (e.g. >>>>> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them >>>>> with an LD_LIBRARY_PATH pointing to the location of the shared >>>>> libraires from these external dependencies, e.g. >>>>> >>>>> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master >>>>> >>>>> It would be nice if we didn't have to explicitly set the >>>>> LD_LIBRARY_PATH to launch these binaries when linking against any >>>>> external shared library dependencies. >>>>> >>>>> One way around this would be to make sure that all external library >>>>> dependencies were stored in standard search paths for the dynamic >>>>> linker. This is typically what happens if you install these >>>>> dependencies via a standard package manager (e.g. apt-get, yum, etc.). >>>>> Sometimes this is undesirable (or impossible) though, especially if >>>>> the external dependencies do not exist as packages or follow a >>>>> non-standard directory hierarchy in terms of where it places its >>>>> include files, libraries, etc. >>>>> >>>>> Another option is to install the paths to these external libraries >>>>> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the >>>>> dynamic linker will search them at runtime. This is also unfeasible >>>>> at times and has the added disadvantage that these library paths will >>>>> now be searched for *all* binaries that get executed (not just the >>>>> ones we currently care about). >>>>> >>>>> The final option (and the one I'm proposing here) is to set the >>>>> 'rpath' of the binary to point to the location of the external shared >>>>> library on the build machine. The rpath is embedded into the binary >>>>> at link time and is used to give the linker an extra set of paths to >>>>> search for shared libraries at runtime. This obvious advantage here >>>>> is that setting rpath allows us to run our binaries without requiring >>>>> LD_LIBRARY_PATH or any of the other methods mentioned above to tell >>>>> the linker where to find our shared libraries. However, it has the >>>>> disadvantage of baking a path into the binary that may only exist on >>>>> the specific machine the binary was built on. >>>>> >>>>> That said, the standard search order used by the dynamic linker to >>>>> find shared libraries is: >>>>> >>>>> 1) LD_LIBRARY_PATH >>>>> 2) rpath >>>>> 4) the ldcache (/etc/ld.so.conf on linux) >>>>> 3) default paths (e.g. /lib, /usr/local/lib) >>>>> >>>>> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH >>>>> if we wanted to. Moreover, we could even change the rpath at the time >>>>> of deployment (e.g. via chrpath on linux). This may be desirable if >>>>> the shared libraries are installed at different locations on the >>>>> deployment machine. >>>>> >>>>> If there are no objections, I therefore propose we modify the >>>>> following files to add rpaths to all external dependencies set via >>>>> --with-* flags: >>>>> >>>>> ./configure.ac >>>>> ./3rdparty/libprocess/3rdparty/stout/configure.ac >>>>> ./3rdparty/libprocess/configure.ac >>>>> >>>>> The pattern would change from: >>>>> >>>>> CPPFLAGS="-I${with_thing}/include $CPPFLAGS" >>>>> LDFLAGS="-L${with_thing}/lib $LDFLAGS" >>>>> >>>>> to include an additional line with: >>>>> >>>>> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS" >>>>> >>>>> I know there has some hesitation with this in the past (especially >>>>> when it comes to producing rpms or debs, where baking in an rpath >>>>> seems really strange), but I'm hoping people will agree that it's >>>>> useful enough that it makes sense to include the rpaths as the default >>>>> case. We can always run a post-processing step to strip them in cases >>>>> where they are undesirable. >>>>> >>>>> Thanks! >>>>> >>>>> -- >>>>> ~Kevin >>>> >>>> >>>> >>>> -- >>>> ~Kevin >>> >> >> >> >> -- >> ~Kevin > > > > -- > ~Kevin -- ~Kevin
