Francis Chuang created MESOS-3462:
-------------------------------------
Summary: Containerization issues with mesos running on CoreOS
Key: MESOS-3462
URL: https://issues.apache.org/jira/browse/MESOS-3462
Project: Mesos
Issue Type: Bug
Components: containerization
Affects Versions: 0.24.0
Environment: CoreOS 801.0.0 64-bit
Reporter: Francis Chuang
These are the steps to I used to build mesos 0.24.0 on Ubuntu 15.04 64-bit:
wget http://www.apache.org/dist/mesos/0.24.0/mesos-0.24.0.tar.gz
wget http://mirror.ventraip.net.au/apache/apr/apr-1.5.2.tar.gz
wget http://mirror.ventraip.net.au/apache/apr/apr-util-1.5.4.tar.gz
wget http://mirror.ventraip.net.au/apache/subversion/subversion-1.9.0.tar.gz
wget http://www.sqlite.org/sqlite-amalgamation-3071501.zip
wget ftp://ftp.cyrusimap.org/cyrus-sasl/cyrus-sasl-2.1.26.tar.gz
mkdir /tmp/mesos-build
cd /tmp/mesos-build
# Build apr
tar zxf apr-$APR_VERSION.tar.gz
cd apr-$APR_VERSION
./configure CC=gcc-4.8 --prefix=/tmp/mesos-build/apr
make
make install
cd ..
# Build apr-util
tar zxf apr-util-$APR_UTIL_VERSION.tar.gz
cd apr-util-$APR_UTIL_VERSION
./configure CC=gcc-4.8 --prefix=/tmp/mesos-build/apr-util
--with-apr=/tmp/mesos-build/apr
make
make install
cd ..
# Build libsasl2
tar zxf cyrus-sasl-$SASL_VERSION.tar.gz
cd cyrus-sasl-$SASL_VERSION
./configure CC=gcc-4.8 CPPFLAGS=-I/usr/include/openssl
--prefix=/tmp/mesos-build/sasl2 --enable-cram
make
make install
cd ..
# Build subversion
tar zxf subversion-$SVN_VERSION.tar.gz
unzip sqlite-amalgamation-$SQLITE_AMALGATION_VERSION.zip
mv sqlite-amalgamation-$SQLITE_AMALGATION_VERSION/
subversion-$SVN_VERSION/sqlite-amalgamation/
cd subversion-$SVN_VERSION
./configure CC=gcc-4.8 CXX=g++-4.8 --prefix=/tmp/mesos-build/svn
--with-apr=/tmp/mesos-build/apr --with-apr-util=/tmp/mesos-build/apr-util
--with-sasl=/tmp/mesos-build/sasl2
make
make install
cd ..
# Build curl
tar zxf curl-$CURL_VERSION.tar.gz
cd curl-$CURL_VERSION
./configure CC=gcc-4.8 --prefix=/tmp/mesos-build/curl
make
make install
cd ..
# Build mesos
tar zxf mesos-$MESOS_VERSION.tar.gz
cd mesos-$MESOS_VERSION
mkdir build
cd build
../configure CC=gcc-4.8 CXX=g++-4.8 LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos
--with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr
--with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
make
make install
cd ..
cd ..
# Copy shared objects into mesos build
cp apr/lib/libapr-1.so.0.5.2 mesos/lib/libapr-1.so.0
cp apr-util/lib/libaprutil-1.so.0.5.4 mesos/lib/libaprutil-1.so.0
cp sasl2/lib/libsasl2.so.3.0.0 mesos/lib/libsasl2.so.3
cp svn/lib/libsvn_delta-1.so.0.0.0 mesos/lib/libsvn_delta-1.so.0
cp svn/lib/libsvn_subr-1.so.0.0.0 mesos/lib/libsvn_subr-1.so.0
I then compress the build into an archive and distributed it onto my CoreOS
nodes.
Once I have the archive extracted on each node, I start the master and slaves:
/opt/mesos/sbin/mesos-master --zk=zk://192.168.33.10/mesos --quorum=1
--hostname=192.168.33.10 --ip=192.168.33.10
--webui_dir=/opt/mesos/share/mesos/webui --cluster=mesos
/opt/mesos/sbin/mesos-slave --hostname=192.168.33.11 --ip=192.168.33.11
--master=zk://192.168.33.10/mesos
--executor_environment_variables='{"LD_LIBRARY_PATH": "/opt/mesos/lib", "PATH":
"/opt/java/bin:/usr/sbin:/usr/bin"}' --containerizers=docker,mesos
--executor_registration_timeout=60mins --launcher_dir=/opt/mesos/libexec/mesos/
In addition, the following environment variables are set:
LD_LIBRARY_PATH=/opt/mesos/lib/
JAVA_HOME=/opt/java
MESOS_NATIVE_JAVA_LIBRARY=/opt/mesos/lib/libmesos.so
I am finding that when I run meso-hdfs from https://github.com/mesosphere/hdfs,
the scheduler starts properly and launches the executors. However, the
executors will fail and terminate without writing any error to stderr and
stdout.
I have reproduced the same problem with mesos 0.24, 0.23 and 0.22.1
If I install mesos onto a Ubuntu machine (tried 14.04 and 15.04 64-bit) using
the apt-repositories, this problem does not happen.
I am not well-versed with mesos internals, but it was pointed out that it's
most likely a containerization issue. This issue on github documents the
process of trying to get mesos-hdfs to work on my compiled mesos binaries:
https://github.com/mesosphere/hdfs/issues/194
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
