Re: Building portable binaries

[CCAAT]

On 09/19/2015 07:06 PM, F21 wrote:

Hey James,

Thanks for sharing the ebuild! I spun up gentoo as a docker container
and copied the ebuild to
/usr/local/portage/clustering/mesos/mesos-0.22.0.ebuild.


Hello F21,

First, apologies for not setting up a more appropriate repo. Github
is on my 'todo' list. I code command line and work mostly on
isolated systems, as an old school coder.

I'm no VM nor container concierge  so for those types of setups, you are 
going to have to find expertise, documents and such elsewhere. You would 
be best advise to join the gentoo-user community to access a very deep 
and wide bench of experts, all centric to gentoo. There are numerous 
forums and irc channels too.



Traditionally, Gentoo puts the portage tree into /usr/portage. The 
ebuilds tarballs are downloaded to /usr/portage/distfiles for raw 
storage by the ebuild. Some are now using /var/portage/.


I use /usr/local/portage to hack new ebuilds before allowing others
to test. With gentoo you can put things where you like, but some 
adjustments need to be made. When I roll all of this out it will be

much easier to just point to an overlay and install from there. But,
I have yet to put an overlay up. Here is one for zookeeper::



[U] sys-cluster/zookeeper
 Available versions:  (~)3.4.6^mb[3] (~)3.4.6[2] (~)3.4.6^mb[5] 
(~)3.4.6-r1^mb[1] {ELIBC="FreeBSD" PYTHON_TARGETS="python2_7"}
 Installed versions:  3.4.6^mb[2](02:35:35 PM 
09/19/2015)(ELIBC="-FreeBSD" PYTHON_TARGETS="python2_7")

 Homepage:http://zookeeper.apache.org/
 Description: ZooKeeper is a high-performance coordination 
service for distributed applications.


* sys-cluster/apache-zookeeper [4]
 Available versions:  ~3.3.6^mb {ELIBC="FreeBSD"}
 Homepage:http://hadoop.apache.org/
 Description: ZooKeeper is a high-performance coordination 
service for distributed applications.


[1] "jackslap" /usr/local/portage
[2] "ultrabug" /var/lib/layman/ultrabug
[3] "fw-overlay" layman/fw-overlay
[4] "gentoo-zh" layman/gentoo-zh
[5] "ultrabug" layman/ultrabug


You can see others are hacking on creating a 100% build from source 
solution for clustering. 'jackslap' is my local repo, which I just email 
to you.I also subscribe to ultrabug's overlay repo, using the layman 
tools to installation. Overlays have their own tree locations that can 
vary from actual gentoo maintained servers, to github, to private 
machines for developers, hackers and any other folks that code.
Gentoo has many derivative distros, some roll binaries, others are on a 
rolling release, just like gentoo. Since the 'systemd wars', the fact 
that gentoo has a robust "openrc" alternative to systemd means that

part of our developer community is growing very fast with new talent
and old coders alike. As more folks roll out clusters, kernel tuning,
a lost art for regular users at all major distros, is going to become 
quintessentially important to get the best performance out of your 
cluster/cloud. The way gentoo is set up, you can use an endless variety 
of boot/kernel semantics to fit your needs. We have very robust tools

for kernel performance studies available with gentoo.


I work strictly on real hardware. I'm a 'bare metal' kind of EE hack.
That said, if you are serious about gentoo, my recommendation' would
be to use older hardware until you get everything working and figured
out what you want to do. Then VM, Container, Cloud or get new/cheap 
cluster gear like aarch64 (arm64). YMMV depending on your goals.




However, when I run ebuild mesos-0.22.0.ebuild manifest, it throws an
error about the metadata:

Appending /usr/local/portage to PORTDIR_OVERLAY...
Error(s) in metadata for 'clustering/mesos-0.22.0':
   DEPEND: Invalid atom (.keep), token 12
   RDEPEND: Invalid atom (.keep), token 7


Make sure you do not have simple errors do to command line mistakes
in your copy process. Those errors are explained  more fully in the 
gentoo Devmanual. Depends are codes that are needed to compile your 
mesos. RDEPENDS are runtime dependencies. Some codes are both.



Any ideas why this might be happening?


Gentoo is not for the faint of heart. It is for those robust users
or for folks that really want to learn deeply how things work. The
great news about gentoo, although many will argue negatively, is that
in clusters, you can get 2-10% performance increase by minimizing codes,
particularly stripped and optimized kernels. Over hundreds of CPUs
that's going to be a huge performance difference. Gentoo had a very rich
echo system of distributed processing years ago. Most of those folks 
took there expertise to corporations for exploit and let gentoo's

position in clustering languish. What gentoo needs now more than ever
are java coders, as the team is excellent, but under staffed, atm.

Now there is a very robust team of arm64 devs, a veritable who's who,
working bring gentoo to arm64. Others like myself are more focused on 
the 

Re: Building portable binaries

[F21]

Hey James,

Thanks for sharing the ebuild! I spun up gentoo as a docker container 
and copied the ebuild to 
/usr/local/portage/clustering/mesos/mesos-0.22.0.ebuild.


However, when I run ebuild mesos-0.22.0.ebuild manifest, it throws an 
error about the metadata:


Appending /usr/local/portage to PORTDIR_OVERLAY...
Error(s) in metadata for 'clustering/mesos-0.22.0':
  DEPEND: Invalid atom (.keep), token 12
  RDEPEND: Invalid atom (.keep), token 7

Any ideas why this might be happening?

Cheers!

On 20/09/2015 4:35 AM, CCAAT wrote:

Hey F21,

The ebuild is attached. Hopefully I'll be setting up github for this 
and other ebuilds I have been hacking on. I'm not a dev, but debugging 
these ebuilds in not difficult [1].



Please let me know how this ebuild works for you. I never tested it 
extensively



James




On 09/17/2015 11:09 PM, F21 wrote:

That sounds really interesting! I am just in the process of spinning up
a gentoo vm.

Would you mind sharing your ebuild for mesos-0.22.0 via a gist on 
Github?


On 18/09/2015 12:58 PM, CCAAT wrote:

On 09/17/2015 06:33 PM, F21 wrote:

Is there anyway to build portable binaries for mesos?



You should try out gentoo linux, everything is built from sources.

Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
or less.

Gentoo is designed from the ground up to build form sources. We have
a rich 'cross-compile' environment for things like aarch64; so building
mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.


There a bit of reading, but the gentoo 'devmanual' pretty much guides
you through the process [1]. Gentoo also has a great package manager.
Here is a (very profane) rant/comparison of some common package 
managers

and their inherent weaknesses [2]. If you want to see how simple the
gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
and installs the package, very neatly. And there is lots of help and
encouragement from a long list of talented devs.

Gentoo is not for the weak minded or folks that do not wish to master
the deep details of linux. caveat emptor. CoreOS uses much of gentoo
in it's build/management, if that option helps.


hth,
James





[1] https://devmanual.gentoo.org/


[2]
http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php 

Re: Building portable binaries

[CCAAT]

Hey F21,

The ebuild is attached. Hopefully I'll be setting up github for this and 
other ebuilds I have been hacking on. I'm not a dev, but debugging these 
ebuilds in not difficult [1].



Please let me know how this ebuild works for you. I never tested it 
extensively



James




On 09/17/2015 11:09 PM, F21 wrote:

That sounds really interesting! I am just in the process of spinning up
a gentoo vm.

Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github?

On 18/09/2015 12:58 PM, CCAAT wrote:

On 09/17/2015 06:33 PM, F21 wrote:

Is there anyway to build portable binaries for mesos?



You should try out gentoo linux, everything is built from sources.

Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
or less.

Gentoo is designed from the ground up to build form sources. We have
a rich 'cross-compile' environment for things like aarch64; so building
mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.


There a bit of reading, but the gentoo 'devmanual' pretty much guides
you through the process [1]. Gentoo also has a great package manager.
Here is a (very profane) rant/comparison of some common package managers
and their inherent weaknesses [2]. If you want to see how simple the
gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
and installs the package, very neatly. And there is lots of help and
encouragement from a long list of talented devs.

Gentoo is not for the weak minded or folks that do not wish to master
the deep details of linux. caveat emptor. CoreOS uses much of gentoo
in it's build/management, if that option helps.


hth,
James





[1] https://devmanual.gentoo.org/


[2]
http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php






# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
## hack by James aug 2014

EAPI=5

inherit autotools eutils flag-o-matic multilib

MY_PV=${PV/_/}

DESCRIPTION="fast cluster manager for distributed applications"
HOMEPAGE="http://mesos.apache.org/;

SRC_URI="http://apache.org/dist/${PN}/${PV}/${P}.tar.gz;

LICENSE="Apache-2.0"
KEYWORDS="~amd64"
IUSE="java python"
SLOT="0"

DEPEND="net-misc/curl
dev-libs/cyrus-sasl
python? ( dev-lang/python dev-python/boto )
java? ( virtual/jdk )
dev-java/maven-bin 
dev-libs/hyperleveldb
dev-python/pip
dev-python/wheel
dev-vcs/subversion"

RDEPEND="python? ( dev-lang/python )
 >=virtual/jdk-1.6
 dev-java/maven-bin 
 ${DEPEND}"


S="${WORKDIR}/${P}"

ECONF_SOURCE="${S}"

src_prepare() {
mkdir "${S}/build" || die
}

src_configure() {
cd "${S}/build"
econf \
$(use_enable python) \
$(use_enable java)
}

src_compile() {
cd "${S}/build"
emake -j1 V=1
}

src_install() {
cd "${S}/build"
emake DESTDIR="${D}" install || die "emake install failed"
}

Re: Building portable binaries

[Alexander Gallego]

My build system is a HUGE PITA.

Right now it is a set of 77 ansible roles. For configuring. half of them
are for mesos.

The reason to have ansible do it exactly because of what you are seeing,
but it gets more complex for framework authors (combination of operating
systems types, versions, libraries, system installed libs, etc)

We build all the transitive dependencies from source too (as many as we can)

So, the flags are exactly what you're using and adding
--with-glog=/usr/local/lib

We just  ` make `  and the ansible scripts move things to the right
directories.

Basically we found that trying to give all of the transitive dependencies a
custom installation path and custom LD_LIBRARY_PATH was hard and it kept
breaking.

So, the way we build it is:

1. Only run make for all deps (never make install - though, some libs
actually perform an install with make, just FYI)
2. Manually link or move the libraries to /usr/local/lib
3. fix your ldconfig


Packaging contents for my deployments is simpler as I deploy only on docker
images. Effectively:


```

 curl -L https:.. | tar -xzf -

```

And it's not an issue because we've placed things in the default
directories.

- alex









On Thu, Sep 17, 2015 at 8:16 PM, F21  wrote:

> Thanks for your pointers! I will give that a try to see if I can track
> down the problem.
>
> Do you mind sharing a bit more about your build process?
>
> What do your ./configure flags look like? Also, do you do a `make install`
> and then package up the contents?
>
> Cheers!
>
>
> On 18/09/2015 10:06 AM, Alexander Gallego wrote:
>
> I run a custom build of mesos and it works pretty reliably, so I'll try to
> share what I do in the hopes that it helps you.
>
> The symptom you are seeing of exiting before anything is output to the
> logs has been usually an issue with the LD_LIBRARY_PATH being screwed up. I
> would try to ssh into the machines and run the command manually.
>
> Native code is a bit hard to debug, but this is what I do for a cluster to
> figure out mesos issues.
>
> 1. Recompile with -ggdb and ship dwarf symbols (if you don't want to,
> you'll have to symbolize the binary later)
> 2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your
> machines
> 3. wrap your mesos executors (if native) as well as mesos-master and
> mesos-slave in a bash script that you control
> 4. In this bash script, set the filehandle limits from the soft to the hard
> 5. In this bash script, set the core dump limits to unlimited
>
> When it crashes, it will be pretty straight forward to debug with the core
> dump, it will almost tell you the exact assembly instruction, or exception
> in the code.
>
> I've never had crashes with mesos after compiling with
>
> ```
> -mtune=generic
> ```
>
> I had something similar to my rocksdb build (
> 
> https://github.com/facebook/rocksdb/issues/690) happened to my mesos
> build when I was first starting and compiling with -mtune=native
>
> If you are linking the libmesos.so into another process (i.e.: you wrote
> your own native executor) then make sure you do something similar to the
> core dump limits.
>
> Note: If you are running dockerized/containerized deployments you won't be
> able to modify the /proc/sys/* , so this will have to be done on the host
> computers.
>
> I've been meaning to write a blog post about this, but this is the TL;DR.
>
> Hope this helps.
>
> - Alex
>
>
>
>
>
>
>
> On Thu, Sep 17, 2015 at 7:33 PM, F21  wrote:
>
>> Is there anyway to build portable binaries for mesos?
>>
>> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr
>> and then built mesos using the following:
>>
>> ../configure CC=gcc-4.8 CXX=g++-4.8
>> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
>> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos
>> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr
>> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
>> make
>> make install
>>
>> I then compress /tmp/mesos-build/mesos into an archive and distribute it
>> to my machines. The only problem is that the build seems to be buggy. For
>> example, I've been experiencing a containerization issues where the
>> executors will crash, but not output anything useful to stderr and stdout.
>> See 
>> https://github.com/mesosphere/hdfs/issues/194
>>
>> Is there a definite way to build portable binaries that I can easily copy
>> to another machine to run?
>>
>
>
>
> --
>
>
>
>
>
> Alexander Gallego |   http://concord.io
>
>
>


-- 





Alexander Gallego |  http://concord.io

Re: Building portable binaries

[Alexander Gallego]

I run a custom build of mesos and it works pretty reliably, so I'll try to
share what I do in the hopes that it helps you.

The symptom you are seeing of exiting before anything is output to the logs
has been usually an issue with the LD_LIBRARY_PATH being screwed up. I
would try to ssh into the machines and run the command manually.

Native code is a bit hard to debug, but this is what I do for a cluster to
figure out mesos issues.

1. Recompile with -ggdb and ship dwarf symbols (if you don't want to,
you'll have to symbolize the binary later)
2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your machines
3. wrap your mesos executors (if native) as well as mesos-master and
mesos-slave in a bash script that you control
4. In this bash script, set the filehandle limits from the soft to the hard
5. In this bash script, set the core dump limits to unlimited

When it crashes, it will be pretty straight forward to debug with the core
dump, it will almost tell you the exact assembly instruction, or exception
in the code.

I've never had crashes with mesos after compiling with

```
-mtune=generic
```

I had something similar to my rocksdb build (
https://github.com/facebook/rocksdb/issues/690) happened to my mesos build
when I was first starting and compiling with -mtune=native

If you are linking the libmesos.so into another process (i.e.: you wrote
your own native executor) then make sure you do something similar to the
core dump limits.

Note: If you are running dockerized/containerized deployments you won't be
able to modify the /proc/sys/* , so this will have to be done on the host
computers.

I've been meaning to write a blog post about this, but this is the TL;DR.

Hope this helps.

- Alex







On Thu, Sep 17, 2015 at 7:33 PM, F21  wrote:

> Is there anyway to build portable binaries for mesos?
>
> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr
> and then built mesos using the following:
>
> ../configure CC=gcc-4.8 CXX=g++-4.8
> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos
> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr
> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
> make
> make install
>
> I then compress /tmp/mesos-build/mesos into an archive and distribute it
> to my machines. The only problem is that the build seems to be buggy. For
> example, I've been experiencing a containerization issues where the
> executors will crash, but not output anything useful to stderr and stdout.
> See https://github.com/mesosphere/hdfs/issues/194
>
> Is there a definite way to build portable binaries that I can easily copy
> to another machine to run?
>



-- 





Alexander Gallego |  http://concord.io

Re: Building portable binaries

[F21]

Thanks for your pointers! I will give that a try to see if I can track 
down the problem.


Do you mind sharing a bit more about your build process?

What do your ./configure flags look like? Also, do you do a `make 
install` and then package up the contents?


Cheers!

On 18/09/2015 10:06 AM, Alexander Gallego wrote:
I run a custom build of mesos and it works pretty reliably, so I'll 
try to share what I do in the hopes that it helps you.


The symptom you are seeing of exiting before anything is output to the 
logs has been usually an issue with the LD_LIBRARY_PATH being screwed 
up. I would try to ssh into the machines and run the command manually.


Native code is a bit hard to debug, but this is what I do for a 
cluster to figure out mesos issues.


1. Recompile with -ggdb and ship dwarf symbols (if you don't want to, 
you'll have to symbolize the binary later)
2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your 
machines
3. wrap your mesos executors (if native) as well as mesos-master and 
mesos-slave in a bash script that you control
4. In this bash script, set the filehandle limits from the soft to the 
hard

5. In this bash script, set the core dump limits to unlimited

When it crashes, it will be pretty straight forward to debug with the 
core dump, it will almost tell you the exact assembly instruction, or 
exception in the code.


I've never had crashes with mesos after compiling with

```
-mtune=generic
```

I had something similar to my rocksdb build 
(https://github.com/facebook/rocksdb/issues/690) happened to my mesos 
build when I was first starting and compiling with -mtune=native


If you are linking the libmesos.so into another process (i.e.: you 
wrote your own native executor) then make sure you do something 
similar to the core dump limits.


Note: If you are running dockerized/containerized deployments you 
won't be able to modify the /proc/sys/* , so this will have to be done 
on the host computers.


I've been meaning to write a blog post about this, but this is the TL;DR.

Hope this helps.

- Alex







On Thu, Sep 17, 2015 at 7:33 PM, F21 > wrote:


Is there anyway to build portable binaries for mesos?

Currently, I have tried building my own libsvn, libsasl2, libcurl,
libapr and then built mesos using the following:

../configure CC=gcc-4.8 CXX=g++-4.8
LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2
--prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn
--with-apr=/tmp/mesos-build/apr
--with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
make
make install

I then compress /tmp/mesos-build/mesos into an archive and
distribute it to my machines. The only problem is that the build
seems to be buggy. For example, I've been experiencing a
containerization issues where the executors will crash, but not
output anything useful to stderr and stdout. See
https://github.com/mesosphere/hdfs/issues/194

Is there a definite way to build portable binaries that I can
easily copy to another machine to run?




--





Alexander Gallego | http://concord.io

Re: Building portable binaries

[CCAAT]

First, here is a very cool gentoo vm with btrfs in a raid-one config.

https://docs.google.com/document/d/1VJlJyYLTZScta9a81xgKOIBjYsG3_VfxxmUSxG23Uxg/edit?pli=1

I'll clean up the ebuild and post tomorrow. I got an old spark and 
zookeeper is floating around. I got sidetracked into working on

a rapid install semantic for gentoo


Tomorrow.

James


On 09/17/2015 11:09 PM, F21 wrote:

That sounds really interesting! I am just in the process of spinning up
a gentoo vm.

Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github?

On 18/09/2015 12:58 PM, CCAAT wrote:

On 09/17/2015 06:33 PM, F21 wrote:

Is there anyway to build portable binaries for mesos?



You should try out gentoo linux, everything is built from sources.

Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
or less.

Gentoo is designed from the ground up to build form sources. We have
a rich 'cross-compile' environment for things like aarch64; so building
mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.


There a bit of reading, but the gentoo 'devmanual' pretty much guides
you through the process [1]. Gentoo also has a great package manager.
Here is a (very profane) rant/comparison of some common package managers
and their inherent weaknesses [2]. If you want to see how simple the
gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
and installs the package, very neatly. And there is lots of help and
encouragement from a long list of talented devs.

Gentoo is not for the weak minded or folks that do not wish to master
the deep details of linux. caveat emptor. CoreOS uses much of gentoo
in it's build/management, if that option helps.


hth,
James





[1] https://devmanual.gentoo.org/


[2]
http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php

Re: Building portable binaries

[James Peach]

> On Sep 17, 2015, at 4:33 PM, F21  wrote:
> 
> Is there anyway to build portable binaries for mesos?
> 
> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr and 
> then built mesos using the following:
> 
> ../configure CC=gcc-4.8 CXX=g++-4.8 
> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib 
> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos 
> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr 
> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
> make
> make install
> 
> I then compress /tmp/mesos-build/mesos into an archive and distribute it to 
> my machines. The only problem is that the build seems to be buggy. For 
> example, I've been experiencing a containerization issues where the executors 
> will crash, but not output anything useful to stderr and stdout. See 
> https://github.com/mesosphere/hdfs/issues/194
> 
> Is there a definite way to build portable binaries that I can easily copy to 
> another machine to run?

You could do a statically linked build by doing configure --enable-static 
--disable-shared. I don't know whether that is supported in the Mesos build, 
but it is a standard automake feature, so if it fails it should be fixable.

J