That's interesting because autotools actually does the opposite. It uses a wrapper script to avoid setting the rpath after a build, but then sets the rpath (relative to --prefix) during 'make install'.
I know there has been some work towards moving to a CMake build, but I'm not sure exactly what the status of that is at the moment. On Thu, Feb 18, 2016 at 6:46 AM, Matthias Bach <[email protected]> wrote: > Hi Kevin, > > CMake will add rpaths during build and then strip them in `make install` > which solves both, the development and the deployment use case. In my > opinion this is the proper way to do it. > > Regards, > Matthias > > Am 15.02.2016 um 00:31 schrieb Kevin Klues: >> I was wrong. This unset the rpath in the binary, but left the rpath >> set in libmesos.so. I am going to leave this as an open issue for >> now, pending a better solution. >> >> On Fri, Feb 12, 2016 at 5:03 PM, Kevin Klues <[email protected]> wrote: >>> Playing with this a bit more, I came up with a solution that bridges >>> the gap between the two alternatives I proposed before. Basically, I >>> found a way to get the wrapper script to set the rpath when running >>> the binary from the build directory, but leave the rpath of the actual >>> binary unchanged. Best of both worlds. >>> >>> Instead of explicity setting the global LDFLAGS in configure.ac, I now >>> set an intermediate variable called LIBMESOS_RPATH for all non-bundled >>> packages: >>> >>> LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_zookeeper}/lib" >>> LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_glog}/lib" >>> ... >>> >>> Then at the bottom of configure.ac, I do: >>> >>> AC_SUBST(LIBMESOS_RPATH, ["${LIBMESOS_RPATH//:/-rpath }"]) >>> >>> This substitutes all colons in LIBMESOS_RPATH with "-rpath " to form a >>> valid string of rpaths for consumption by Makefile.am. I chose to >>> format the original LIBMESOS_RPATH string with colons so that it forms >>> a valid LD_LIBRARY_PATH as well (plus it's easier to tokenize by ':' >>> than by '-rpath '). >>> >>> Then in Makefile.am I do: >>> >>> libmesos_la_LDFLAGS += $(LIBMESOS_RPATH) >>> >>> This sets the rpath for libmesos using *libtool's* LDFLAGS rather than >>> setting the global LDFLAGS for all linked objects (as I was doing >>> before). >>> >>> With this change, I can inspect libtools wrapper scripts to see that indeed >>> it patches up the binary with the added rpaths, but libmesos itself >>> has no rpath set. >>> >>> Pending any objections, a RR will be forthcoming. >>> >>> On Fri, Feb 12, 2016 at 3:31 PM, Kevin Klues <[email protected]> wrote: >>>> I like that idea. Don't set rpath by default, but allow people to >>>> specify that it should be set via a flag. How does >>>> >>>> --set-rpath-for-external-libs >>>> >>>> sound for a name. Too long? I don't like just --with-rpath, because >>>> it's not descriptive enough in my opinion. >>>> >>>> On Fri, Feb 12, 2016 at 2:34 PM, Jojy Varghese <[email protected]> wrote: >>>>> Maybe have an opt-in (say —with-rpath)? >>>>> >>>>> -Jojy >>>>> >>>>>> On Feb 12, 2016, at 2:19 PM, Kevin Klues <[email protected]> wrote: >>>>>> >>>>>> To be clear, I'm actually a bit torn both ways on this. >>>>>> >>>>>> On the one hand, including the rpath makes it easy for those who don't >>>>>> know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass >>>>>> their paths to their external dependencies at configure time and then >>>>>> run their binaries without further effort. >>>>>> >>>>>> On the other hand, maybe they should be cognizant of the fact that >>>>>> something is going on under the hood to actually allow their binaries >>>>>> to link properly (i.e. I can imagine a situation where someone builds, >>>>>> runs, and tests everything locally, and then is confused as to why it >>>>>> nothing works once deployed). >>>>>> >>>>>> In my previous email, I argue that we should include the rpath by >>>>>> default, and can strip it later if we don't want it for some reason >>>>>> (i.e. when bundling into debs/rpms). Conversely, we could leave it >>>>>> out by default and only set it as a post-processing step in situations >>>>>> where we actually care about it. >>>>>> >>>>>> I'm curious what other people's thoughts are. >>>>>> >>>>>> >>>>>> On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <[email protected]> wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> A discussion came up recently around including rpaths in our mesos >>>>>>> binaries to help resolve any shared library dependencies that don't >>>>>>> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.). >>>>>>> >>>>>>> By default, there are no shared library dependencies that exist in >>>>>>> non-standard paths, because we bundle all of these dependencies into >>>>>>> the mesos source and statically link them into our executables (e.g. >>>>>>> glog, zookeeper, etc.) >>>>>>> >>>>>>> However, if you configure mesos with e.g. >>>>>>> >>>>>>> ../configure ----disable-bundled >>>>>>> >>>>>>> or the more selective >>>>>>> >>>>>>> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ... >>>>>>> >>>>>>> then mesos will be built with an external shared library dependency >>>>>>> (e.g. glog and zookeeper in this case). >>>>>>> >>>>>>> The build system is smart enough to set up LDFLAGS so we can link >>>>>>> against whatever external libraries are passed in via the --with-* >>>>>>> flags. >>>>>>> >>>>>>> However, when we go to run the binaries that are produced (e.g. >>>>>>> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them >>>>>>> with an LD_LIBRARY_PATH pointing to the location of the shared >>>>>>> libraires from these external dependencies, e.g. >>>>>>> >>>>>>> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master >>>>>>> >>>>>>> It would be nice if we didn't have to explicitly set the >>>>>>> LD_LIBRARY_PATH to launch these binaries when linking against any >>>>>>> external shared library dependencies. >>>>>>> >>>>>>> One way around this would be to make sure that all external library >>>>>>> dependencies were stored in standard search paths for the dynamic >>>>>>> linker. This is typically what happens if you install these >>>>>>> dependencies via a standard package manager (e.g. apt-get, yum, etc.). >>>>>>> Sometimes this is undesirable (or impossible) though, especially if >>>>>>> the external dependencies do not exist as packages or follow a >>>>>>> non-standard directory hierarchy in terms of where it places its >>>>>>> include files, libraries, etc. >>>>>>> >>>>>>> Another option is to install the paths to these external libraries >>>>>>> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the >>>>>>> dynamic linker will search them at runtime. This is also unfeasible >>>>>>> at times and has the added disadvantage that these library paths will >>>>>>> now be searched for *all* binaries that get executed (not just the >>>>>>> ones we currently care about). >>>>>>> >>>>>>> The final option (and the one I'm proposing here) is to set the >>>>>>> 'rpath' of the binary to point to the location of the external shared >>>>>>> library on the build machine. The rpath is embedded into the binary >>>>>>> at link time and is used to give the linker an extra set of paths to >>>>>>> search for shared libraries at runtime. This obvious advantage here >>>>>>> is that setting rpath allows us to run our binaries without requiring >>>>>>> LD_LIBRARY_PATH or any of the other methods mentioned above to tell >>>>>>> the linker where to find our shared libraries. However, it has the >>>>>>> disadvantage of baking a path into the binary that may only exist on >>>>>>> the specific machine the binary was built on. >>>>>>> >>>>>>> That said, the standard search order used by the dynamic linker to >>>>>>> find shared libraries is: >>>>>>> >>>>>>> 1) LD_LIBRARY_PATH >>>>>>> 2) rpath >>>>>>> 4) the ldcache (/etc/ld.so.conf on linux) >>>>>>> 3) default paths (e.g. /lib, /usr/local/lib) >>>>>>> >>>>>>> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH >>>>>>> if we wanted to. Moreover, we could even change the rpath at the time >>>>>>> of deployment (e.g. via chrpath on linux). This may be desirable if >>>>>>> the shared libraries are installed at different locations on the >>>>>>> deployment machine. >>>>>>> >>>>>>> If there are no objections, I therefore propose we modify the >>>>>>> following files to add rpaths to all external dependencies set via >>>>>>> --with-* flags: >>>>>>> >>>>>>> ./configure.ac >>>>>>> ./3rdparty/libprocess/3rdparty/stout/configure.ac >>>>>>> ./3rdparty/libprocess/configure.ac >>>>>>> >>>>>>> The pattern would change from: >>>>>>> >>>>>>> CPPFLAGS="-I${with_thing}/include $CPPFLAGS" >>>>>>> LDFLAGS="-L${with_thing}/lib $LDFLAGS" >>>>>>> >>>>>>> to include an additional line with: >>>>>>> >>>>>>> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS" >>>>>>> >>>>>>> I know there has some hesitation with this in the past (especially >>>>>>> when it comes to producing rpms or debs, where baking in an rpath >>>>>>> seems really strange), but I'm hoping people will agree that it's >>>>>>> useful enough that it makes sense to include the rpaths as the default >>>>>>> case. We can always run a post-processing step to strip them in cases >>>>>>> where they are undesirable. >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> -- >>>>>>> ~Kevin >>>>>> >>>>>> >>>>>> -- >>>>>> ~Kevin >>>> >>>> >>>> -- >>>> ~Kevin >>> >>> >>> -- >>> ~Kevin >> >> > > -- > > Dr. Matthias Bach > Senior Software Engineer > *Blue Yonder GmbH* > Ohiostraße 8 > D-76149 Karlsruhe > > Tel +49 (0)721 383 117 6244 > Fax +49 (0)721 383 117 69 > > [email protected] <mailto:[email protected]> > www.blue-yonder.com <http://www.blue-yonder.com/> > Registergericht Mannheim, HRB 704547 > USt-IdNr. DE DE 277 091 535 > Geschäftsführer: Jochen Bossert, Uwe Weiss (CEO) > -- ~Kevin
