My build system is a HUGE PITA. Right now it is a set of 77 ansible roles. For configuring. half of them are for mesos.
The reason to have ansible do it exactly because of what you are seeing, but it gets more complex for framework authors (combination of operating systems types, versions, libraries, system installed libs, etc) We build all the transitive dependencies from source too (as many as we can) So, the flags are exactly what you're using and adding --with-glog=/usr/local/lib We just ` make ` and the ansible scripts move things to the right directories. Basically we found that trying to give all of the transitive dependencies a custom installation path and custom LD_LIBRARY_PATH was hard and it kept breaking. So, the way we build it is: 1. Only run make for all deps (never make install - though, some libs actually perform an install with make, just FYI) 2. Manually link or move the libraries to /usr/local/lib 3. fix your ldconfig Packaging contents for my deployments is simpler as I deploy only on docker images. Effectively: ``` curl -L https:...... | tar -xzf - ``` And it's not an issue because we've placed things in the default directories. - alex On Thu, Sep 17, 2015 at 8:16 PM, F21 <f21.gro...@gmail.com> wrote: > Thanks for your pointers! I will give that a try to see if I can track > down the problem. > > Do you mind sharing a bit more about your build process? > > What do your ./configure flags look like? Also, do you do a `make install` > and then package up the contents? > > Cheers! > > > On 18/09/2015 10:06 AM, Alexander Gallego wrote: > > I run a custom build of mesos and it works pretty reliably, so I'll try to > share what I do in the hopes that it helps you. > > The symptom you are seeing of exiting before anything is output to the > logs has been usually an issue with the LD_LIBRARY_PATH being screwed up. I > would try to ssh into the machines and run the command manually. > > Native code is a bit hard to debug, but this is what I do for a cluster to > figure out mesos issues. > > 1. Recompile with -ggdb and ship dwarf symbols (if you don't want to, > you'll have to symbolize the binary later) > 2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your > machines > 3. wrap your mesos executors (if native) as well as mesos-master and > mesos-slave in a bash script that you control > 4. In this bash script, set the filehandle limits from the soft to the hard > 5. In this bash script, set the core dump limits to unlimited > > When it crashes, it will be pretty straight forward to debug with the core > dump, it will almost tell you the exact assembly instruction, or exception > in the code. > > I've never had crashes with mesos after compiling with > > ``` > -mtune=generic > ``` > > I had something similar to my rocksdb build ( > <https://github.com/facebook/rocksdb/issues/690> > https://github.com/facebook/rocksdb/issues/690) happened to my mesos > build when I was first starting and compiling with -mtune=native > > If you are linking the libmesos.so into another process (i.e.: you wrote > your own native executor) then make sure you do something similar to the > core dump limits. > > Note: If you are running dockerized/containerized deployments you won't be > able to modify the /proc/sys/* , so this will have to be done on the host > computers. > > I've been meaning to write a blog post about this, but this is the TL;DR. > > Hope this helps. > > - Alex > > > > > > > > On Thu, Sep 17, 2015 at 7:33 PM, F21 <f21.gro...@gmail.com> wrote: > >> Is there anyway to build portable binaries for mesos? >> >> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr >> and then built mesos using the following: >> >> ../configure CC=gcc-4.8 CXX=g++-4.8 >> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib >> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos >> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr >> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl >> make >> make install >> >> I then compress /tmp/mesos-build/mesos into an archive and distribute it >> to my machines. The only problem is that the build seems to be buggy. For >> example, I've been experiencing a containerization issues where the >> executors will crash, but not output anything useful to stderr and stdout. >> See <https://github.com/mesosphere/hdfs/issues/194> >> https://github.com/mesosphere/hdfs/issues/194 >> >> Is there a definite way to build portable binaries that I can easily copy >> to another machine to run? >> > > > > -- > > > > > > Alexander Gallego | <http://concord.io>http://concord.io > > > -- Alexander Gallego | http://concord.io
