I like that idea. Don't set rpath by default, but allow people to specify that it should be set via a flag. How does
--set-rpath-for-external-libs sound for a name. Too long? I don't like just --with-rpath, because it's not descriptive enough in my opinion. On Fri, Feb 12, 2016 at 2:34 PM, Jojy Varghese <[email protected]> wrote: > Maybe have an opt-in (say —with-rpath)? > > -Jojy > >> On Feb 12, 2016, at 2:19 PM, Kevin Klues <[email protected]> wrote: >> >> To be clear, I'm actually a bit torn both ways on this. >> >> On the one hand, including the rpath makes it easy for those who don't >> know anything about LD_LIBRARY_PATH, the ldcache, etc. to simply pass >> their paths to their external dependencies at configure time and then >> run their binaries without further effort. >> >> On the other hand, maybe they should be cognizant of the fact that >> something is going on under the hood to actually allow their binaries >> to link properly (i.e. I can imagine a situation where someone builds, >> runs, and tests everything locally, and then is confused as to why it >> nothing works once deployed). >> >> In my previous email, I argue that we should include the rpath by >> default, and can strip it later if we don't want it for some reason >> (i.e. when bundling into debs/rpms). Conversely, we could leave it >> out by default and only set it as a post-processing step in situations >> where we actually care about it. >> >> I'm curious what other people's thoughts are. >> >> >> On Fri, Feb 12, 2016 at 1:47 PM, Kevin Klues <[email protected]> wrote: >>> Hi all, >>> >>> A discussion came up recently around including rpaths in our mesos >>> binaries to help resolve any shared library dependencies that don't >>> exist in standard library paths (e.g. /lib, /usr/local/lib, etc.). >>> >>> By default, there are no shared library dependencies that exist in >>> non-standard paths, because we bundle all of these dependencies into >>> the mesos source and statically link them into our executables (e.g. >>> glog, zookeeper, etc.) >>> >>> However, if you configure mesos with e.g. >>> >>> ../configure ----disable-bundled >>> >>> or the more selective >>> >>> ../configure --with-glog[=DIR] --with-zookeeper[=DIR] ... >>> >>> then mesos will be built with an external shared library dependency >>> (e.g. glog and zookeeper in this case). >>> >>> The build system is smart enough to set up LDFLAGS so we can link >>> against whatever external libraries are passed in via the --with-* >>> flags. >>> >>> However, when we go to run the binaries that are produced (e.g. >>> mesos-master, mesos-slave, mesos-test, etc.), we have to prefix them >>> with an LD_LIBRARY_PATH pointing to the location of the shared >>> libraires from these external dependencies, e.g. >>> >>> LD_LIBRARY_PATH="/glog/lib:/zookeeper/lib" ./mesos-master >>> >>> It would be nice if we didn't have to explicitly set the >>> LD_LIBRARY_PATH to launch these binaries when linking against any >>> external shared library dependencies. >>> >>> One way around this would be to make sure that all external library >>> dependencies were stored in standard search paths for the dynamic >>> linker. This is typically what happens if you install these >>> dependencies via a standard package manager (e.g. apt-get, yum, etc.). >>> Sometimes this is undesirable (or impossible) though, especially if >>> the external dependencies do not exist as packages or follow a >>> non-standard directory hierarchy in terms of where it places its >>> include files, libraries, etc. >>> >>> Another option is to install the paths to these external libraries >>> into the ldcache (e.g. via /etc/ld.so.conf on linux) so that the >>> dynamic linker will search them at runtime. This is also unfeasible >>> at times and has the added disadvantage that these library paths will >>> now be searched for *all* binaries that get executed (not just the >>> ones we currently care about). >>> >>> The final option (and the one I'm proposing here) is to set the >>> 'rpath' of the binary to point to the location of the external shared >>> library on the build machine. The rpath is embedded into the binary >>> at link time and is used to give the linker an extra set of paths to >>> search for shared libraries at runtime. This obvious advantage here >>> is that setting rpath allows us to run our binaries without requiring >>> LD_LIBRARY_PATH or any of the other methods mentioned above to tell >>> the linker where to find our shared libraries. However, it has the >>> disadvantage of baking a path into the binary that may only exist on >>> the specific machine the binary was built on. >>> >>> That said, the standard search order used by the dynamic linker to >>> find shared libraries is: >>> >>> 1) LD_LIBRARY_PATH >>> 2) rpath >>> 4) the ldcache (/etc/ld.so.conf on linux) >>> 3) default paths (e.g. /lib, /usr/local/lib) >>> >>> Meaning that we could always overwrite the rpath using LD_LIBARY_PATH >>> if we wanted to. Moreover, we could even change the rpath at the time >>> of deployment (e.g. via chrpath on linux). This may be desirable if >>> the shared libraries are installed at different locations on the >>> deployment machine. >>> >>> If there are no objections, I therefore propose we modify the >>> following files to add rpaths to all external dependencies set via >>> --with-* flags: >>> >>> ./configure.ac >>> ./3rdparty/libprocess/3rdparty/stout/configure.ac >>> ./3rdparty/libprocess/configure.ac >>> >>> The pattern would change from: >>> >>> CPPFLAGS="-I${with_thing}/include $CPPFLAGS" >>> LDFLAGS="-L${with_thing}/lib $LDFLAGS" >>> >>> to include an additional line with: >>> >>> LDFLAGS="-Wl,-rpath,${with_thing}/lib $LDFLAGS" >>> >>> I know there has some hesitation with this in the past (especially >>> when it comes to producing rpms or debs, where baking in an rpath >>> seems really strange), but I'm hoping people will agree that it's >>> useful enough that it makes sense to include the rpaths as the default >>> case. We can always run a post-processing step to strip them in cases >>> where they are undesirable. >>> >>> Thanks! >>> >>> -- >>> ~Kevin >> >> >> >> -- >> ~Kevin > -- ~Kevin
