Hi Kevin, CMake will add rpaths during build and then strip them in `make install` which solves both, the development and the deployment use case. In my opinion this is the proper way to do it.
Regards, Matthias Am 15.02.2016 um 00:31 schrieb Kevin Klues: > I was wrong. This unset the rpath in the binary, but left the rpath > set in libmesos.so. I am going to leave this as an open issue for > now, pending a better solution. > > On Fri, Feb 12, 2016 at 5:03 PM, Kevin Klues <[email protected]> wrote: >> Playing with this a bit more, I came up with a solution that bridges >> the gap between the two alternatives I proposed before. Basically, I >> found a way to get the wrapper script to set the rpath when running >> the binary from the build directory, but leave the rpath of the actual >> binary unchanged. Best of both worlds. >> >> Instead of explicity setting the global LDFLAGS in configure.ac, I now >> set an intermediate variable called LIBMESOS_RPATH for all non-bundled >> packages: >> >> LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_zookeeper}/lib" >> LIBMESOS_RPATH="$LIBMESOS_RPATH:${with_glog}/lib" >> ... >> >> Then at the bottom of configure.ac, I do: >> >> AC_SUBST(LIBMESOS_RPATH, ["${LIBMESOS_RPATH//:/-rpath }"]) >> >> This substitutes all colons in LIBMESOS_RPATH with "-rpath " to form a >> valid string of rpaths for consumption by Makefile.am. I chose to >> format the original LIBMESOS_RPATH string with colons so that it forms >> a valid LD_LIBRARY_PATH as well (plus it's easier to tokenize by ':' >> than by '-rpath '). >> >> Then in Makefile.am I do: >> >> libmesos_la_LDFLAGS += $(LIBMESOS_RPATH) >> >> This sets the rpath for libmesos using *libtool's* LDFLAGS rather than >> setting the global LDFLAGS for all linked objects (as I was doing >> before). >> >> With this change, I can inspect libtools wrapper scripts to see that indeed >> it patches up the binary with the added rpaths, but libmesos itself >> has no rpath set. >> >> Pending any objections, a RR will be forthcoming. >> >> On Fri, Feb 12, 2016 at 3:31 PM, Kevin Klues <[email protected]> wrote: >>> I like that idea. Don't set rpath by default, but allow people to >>> specify that it should be set via a flag. How does >>> >>> --set-rpath-for-external-libs >>> >>> sound for a name. Too long? I don't like just --with-rpath, because >>> it's not descriptive enough in my opinion. >>> >>> On Fri, Feb 12, 2016 at 2:34 PM, Jojy Varghese <[email protected]> wrote: >>>> Maybe have an opt-in (say —with-rpath)? >>>> >>>> -Jojy >>>> >>>>> On Feb 12, 2016, at 2:19 PM, Kevin Klues <[email protected]> wrote: >>>>> >>>>> To be clear, I'm actually a bit torn both ways on this. >>>>> >>>>> On the one hand, including the rpath makes it easy for those who don't >>>>> know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass >>>>> their paths to their external dependencies at configure time and then >>>>> run their binaries without further effort. >>>>> >>>>> On the other hand, maybe they should be cognizant of the fact that >>>>> something is going on under the hood to actually allow their binaries >>>>> to link properly (i.e. I can imagine a situation where someone builds, >>>>> runs, and tests everything locally, and then is confused as to why it >>>>> nothing works once deployed). >>>>> >>>>> In my previous email, I argue that we should include the rpath by >>>>> default, and can strip it later if we don't want it for some reason >>>>> (i.e. when bundling into debs/rpms). Conversely, we could leave it >>>>> out by default and only set it as a post-processing step in situations >>>>> where we actually care about it. >>>>> >>>>> I'm curious what other people's thoughts are. >>>>> >>>>> >>>>> On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <[email protected]> wrote: >>>>>> Hi all, >>>>>> >>>>>> A discussion came up recently around including rpaths in our mesos >>>>>> binaries to help resolve any shared library dependencies that don't >>>>>> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.). >>>>>> >>>>>> By default, there are no shared library dependencies that exist in >>>>>> non-standard paths, because we bundle all of these dependencies into >>>>>> the mesos source and statically link them into our executables (e.g. >>>>>> glog, zookeeper, etc.) >>>>>> >>>>>> However, if you configure mesos with e.g. >>>>>> >>>>>> ../configure ----disable-bundled >>>>>> >>>>>> or the more selective >>>>>> >>>>>> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ... >>>>>> >>>>>> then mesos will be built with an external shared library dependency >>>>>> (e.g. glog and zookeeper in this case). >>>>>> >>>>>> The build system is smart enough to set up LDFLAGS so we can link >>>>>> against whatever external libraries are passed in via the --with-* >>>>>> flags. >>>>>> >>>>>> However, when we go to run the binaries that are produced (e.g. >>>>>> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them >>>>>> with an LD_LIBRARY_PATH pointing to the location of the shared >>>>>> libraires from these external dependencies, e.g. >>>>>> >>>>>> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master >>>>>> >>>>>> It would be nice if we didn't have to explicitly set the >>>>>> LD_LIBRARY_PATH to launch these binaries when linking against any >>>>>> external shared library dependencies. >>>>>> >>>>>> One way around this would be to make sure that all external library >>>>>> dependencies were stored in standard search paths for the dynamic >>>>>> linker. This is typically what happens if you install these >>>>>> dependencies via a standard package manager (e.g. apt-get, yum, etc.). >>>>>> Sometimes this is undesirable (or impossible) though, especially if >>>>>> the external dependencies do not exist as packages or follow a >>>>>> non-standard directory hierarchy in terms of where it places its >>>>>> include files, libraries, etc. >>>>>> >>>>>> Another option is to install the paths to these external libraries >>>>>> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the >>>>>> dynamic linker will search them at runtime. This is also unfeasible >>>>>> at times and has the added disadvantage that these library paths will >>>>>> now be searched for *all* binaries that get executed (not just the >>>>>> ones we currently care about). >>>>>> >>>>>> The final option (and the one I'm proposing here) is to set the >>>>>> 'rpath' of the binary to point to the location of the external shared >>>>>> library on the build machine. The rpath is embedded into the binary >>>>>> at link time and is used to give the linker an extra set of paths to >>>>>> search for shared libraries at runtime. This obvious advantage here >>>>>> is that setting rpath allows us to run our binaries without requiring >>>>>> LD_LIBRARY_PATH or any of the other methods mentioned above to tell >>>>>> the linker where to find our shared libraries. However, it has the >>>>>> disadvantage of baking a path into the binary that may only exist on >>>>>> the specific machine the binary was built on. >>>>>> >>>>>> That said, the standard search order used by the dynamic linker to >>>>>> find shared libraries is: >>>>>> >>>>>> 1) LD_LIBRARY_PATH >>>>>> 2) rpath >>>>>> 4) the ldcache (/etc/ld.so.conf on linux) >>>>>> 3) default paths (e.g. /lib, /usr/local/lib) >>>>>> >>>>>> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH >>>>>> if we wanted to. Moreover, we could even change the rpath at the time >>>>>> of deployment (e.g. via chrpath on linux). This may be desirable if >>>>>> the shared libraries are installed at different locations on the >>>>>> deployment machine. >>>>>> >>>>>> If there are no objections, I therefore propose we modify the >>>>>> following files to add rpaths to all external dependencies set via >>>>>> --with-* flags: >>>>>> >>>>>> ./configure.ac >>>>>> ./3rdparty/libprocess/3rdparty/stout/configure.ac >>>>>> ./3rdparty/libprocess/configure.ac >>>>>> >>>>>> The pattern would change from: >>>>>> >>>>>> CPPFLAGS="-I${with_thing}/include $CPPFLAGS" >>>>>> LDFLAGS="-L${with_thing}/lib $LDFLAGS" >>>>>> >>>>>> to include an additional line with: >>>>>> >>>>>> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS" >>>>>> >>>>>> I know there has some hesitation with this in the past (especially >>>>>> when it comes to producing rpms or debs, where baking in an rpath >>>>>> seems really strange), but I'm hoping people will agree that it's >>>>>> useful enough that it makes sense to include the rpaths as the default >>>>>> case. We can always run a post-processing step to strip them in cases >>>>>> where they are undesirable. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> ~Kevin >>>>> >>>>> >>>>> -- >>>>> ~Kevin >>> >>> >>> -- >>> ~Kevin >> >> >> -- >> ~Kevin > > -- Dr. Matthias Bach Senior Software Engineer *Blue Yonder GmbH* Ohiostraße 8 D-76149 Karlsruhe Tel +49 (0)721 383 117 6244 Fax +49 (0)721 383 117 69 [email protected] <mailto:[email protected]> www.blue-yonder.com <http://www.blue-yonder.com/> Registergericht Mannheim, HRB 704547 USt-IdNr. DE DE 277 091 535 Geschäftsführer: Jochen Bossert, Uwe Weiss (CEO)
