Re: Batch/queue frameworks?
I am also interested in something like this, although my requirements are much more simpler. I am interested in a work queue like beanstalkd that will allow me to push to a queue from a web app and have workers to do things like send emails, generate pdfs and resize images. I have thought about running a beanstalkd in a container, but it has some limitations. For example, if it crashes, it needs to be relaunched manually to recover the binlog (which is a no go). Another option I can think of is to use kafka (which has a mesos framework) and have the web app and other parts push jobs into the kafka broker. Workers listening on the broker would pop each job off and execute whatever needs to be done. However, there seems to be a lot of wheel-reinventing what that solution. For example, what if a job depends on another job? There's also a lot of work that needs to be done at a lower level when all I am interested in is to write domain specific code to generate the pdf, resize the image etc. If there's a work queue solution for mesos, I would love to know too. On 7/10/2015 8:08 PM, Brian Candler wrote: On 07/10/2015 09:44, Nikolaos Ballas neXus wrote: Maybe you need to read a bit :) I have read plenty, including those you list, and I didn't find anything which met my requirements. Again I apologise if I was not clear in my question. Spark has a very specific data model (RDDs) and applications which write to its API. I want to run arbitrary compute jobs - think "shell scripts" or "docker containers" which run pre-existing applications which I can't change. And I want to fill a queue or pipeline with those jobs. Hadoop also is for specific workloads, written to run under Hadoop and preferably using HDFS. The nearest Hadoop gets to general-purpose computing, as far as I can see, is its YARN scheduler. YARN can in turn run under Mesos. Therefore a job queue which can run on YARN might be acceptable, although I'd rather not have an additional layer in the stack. (There was an old project for running Torque under YARN, but this has been abandoned) Regards, Brian.
Re: Web ui offers tab blank
794 master.cpp:2739] Processing ACCEPT call for offers: [ 20150926-040939-169978048-5050-594-O1807 ] on slave 20150926-040939-169978048-5050-594-S0 at slave(1)@192.168.33.10:5051 (192.168.33.10) for framework 20150925-063839-169978048-5050-1342- (marathon) at scheduler-4f7b57a5-cded-4e8f-90d7-7ad160488544@192.168.33.10:34340 Even though it's evident there are offers, I still don't see anything in the offers tab of the web ui. On 26/09/2015 4:17 AM, Vinod Kone wrote: What do the master and scheduler logs say? On Fri, Sep 25, 2015 at 5:02 AM, F21 <f21.gro...@gmail.com <mailto:f21.gro...@gmail.com>> wrote: I am running mesos 0.24 on CoreOS. I also have Marathon 0.10.1 running. I am trying to run the HDFS framework and notice that it's complaining that there isn't enough resources when trying to launch an executor. I wanted to see what offers are being made, but when I click the "offers" tab in the web ui, I see no offers. Is this a bug? Or have I neglected a configuration option?
Web ui offers tab blank
I am running mesos 0.24 on CoreOS. I also have Marathon 0.10.1 running. I am trying to run the HDFS framework and notice that it's complaining that there isn't enough resources when trying to launch an executor. I wanted to see what offers are being made, but when I click the "offers" tab in the web ui, I see no offers. Is this a bug? Or have I neglected a configuration option?
Re: Building portable binaries
Hey James, Thanks for sharing the ebuild! I spun up gentoo as a docker container and copied the ebuild to /usr/local/portage/clustering/mesos/mesos-0.22.0.ebuild. However, when I run ebuild mesos-0.22.0.ebuild manifest, it throws an error about the metadata: Appending /usr/local/portage to PORTDIR_OVERLAY... Error(s) in metadata for 'clustering/mesos-0.22.0': DEPEND: Invalid atom (.keep), token 12 RDEPEND: Invalid atom (.keep), token 7 Any ideas why this might be happening? Cheers! On 20/09/2015 4:35 AM, CCAAT wrote: Hey F21, The ebuild is attached. Hopefully I'll be setting up github for this and other ebuilds I have been hacking on. I'm not a dev, but debugging these ebuilds in not difficult [1]. Please let me know how this ebuild works for you. I never tested it extensively James On 09/17/2015 11:09 PM, F21 wrote: That sounds really interesting! I am just in the process of spinning up a gentoo vm. Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github? On 18/09/2015 12:58 PM, CCAAT wrote: On 09/17/2015 06:33 PM, F21 wrote: Is there anyway to build portable binaries for mesos? You should try out gentoo linux, everything is built from sources. Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks or less. Gentoo is designed from the ground up to build form sources. We have a rich 'cross-compile' environment for things like aarch64; so building mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out. There a bit of reading, but the gentoo 'devmanual' pretty much guides you through the process [1]. Gentoo also has a great package manager. Here is a (very profane) rant/comparison of some common package managers and their inherent weaknesses [2]. If you want to see how simple the gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles and installs the package, very neatly. And there is lots of help and encouragement from a long list of talented devs. Gentoo is not for the weak minded or folks that do not wish to master the deep details of linux. caveat emptor. CoreOS uses much of gentoo in it's build/management, if that option helps. hth, James [1] https://devmanual.gentoo.org/ [2] http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
mesos-containerizer: error while loading shared libraries: libmesos-0.24.0.so
I am building mesos 0.24.0 with the goal to compress the binaries into
an archive and distribute them onto other machines to run.
Mesos is configured with the following options and then compiled:
./configure CC=gcc-4.8 CXX=g++-4.8
LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2
--prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn
--with-apr=/tmp/mesos-build/apr --with-sasl=/tmp/mesos-build/sasl2/
--with-curl=/tmp/mesos-build/curl
I then compressed the whole archive and deployed them on to my machines.
Mesos master is launched with the following config:
MESOS_LOG_DIR=/var/log/mesos
MESOS_WORK_DIR=/var/lib/mesos/master
MESOS_CLUSTER=mesos
MESOS_ROLES=mesos_master
MESOS_WEBUI_DIR=/opt/airglow/packages/mesos/share/mesos/webui
MESOS_ZK=zk://192.168.33.10/mesos
MESOS_QUORUM=1
MESOS_HOSTNAME=192.168.33.10
MESOS_IP=192.168.33.10
The slave is started with the following config:
MESOS_LOG_DIR=/var/log/mesos
MESOS_CONTAINERIZERS=docker,mesos
MESOS_EXECUTOR_REGISTRATION_TIMEOUT=60mins
MESOS _HOSTNAME=192.168.33.11
MESOS_IP=192.168.33.11
MESOS_MASTER=zk://192.168.33.10/mesos
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
On all machines I have the following environment variables:
LD_LIBRARY_PATH=/path/to/mesos/lib
JAVA_HOME is also set up correctly
I then attempted to run mesos-hdfs from the master. The scheduler
launches properly.
However, when the scheduler tries to launch a task (such as a journal
node), it fails and I can see this in the stderr of the sandbox:
mesos-containerizer: error while loading shared libraries:
libmesos-0.24.0.so: cannot open shared object file: No such file or
directory
What causes this problem and how can I resolve it?
Re: mesos-containerizer: error while loading shared libraries: libmesos-0.24.0.so
Yes, it exists. I can also see a libmesos.so that is symlinked to
libmesos-0.24.0.so
On 17/09/2015 7:08 PM, haosdent wrote:
Does libmesos-0.24.0.so <http://libmesos-0.24.0.so> exists in
LD_LIBRARY_PATH=/path/to/mesos/lib?
On Thu, Sep 17, 2015 at 5:01 PM, F21 <f21.gro...@gmail.com
<mailto:f21.gro...@gmail.com>> wrote:
I am building mesos 0.24.0 with the goal to compress the binaries
into an archive and distribute them onto other machines to run.
Mesos is configured with the following options and then compiled:
./configure CC=gcc-4.8 CXX=g++-4.8
LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2
--prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn
--with-apr=/tmp/mesos-build/apr
--with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
I then compressed the whole archive and deployed them on to my
machines.
Mesos master is launched with the following config:
MESOS_LOG_DIR=/var/log/mesos
MESOS_WORK_DIR=/var/lib/mesos/master
MESOS_CLUSTER=mesos
MESOS_ROLES=mesos_master
MESOS_WEBUI_DIR=/opt/airglow/packages/mesos/share/mesos/webui
MESOS_ZK=zk://192.168.33.10/mesos <http://192.168.33.10/mesos>
MESOS_QUORUM=1
MESOS_HOSTNAME=192.168.33.10
MESOS_IP=192.168.33.10
The slave is started with the following config:
MESOS_LOG_DIR=/var/log/mesos
MESOS_CONTAINERIZERS=docker,mesos
MESOS_EXECUTOR_REGISTRATION_TIMEOUT=60mins
MESOS _HOSTNAME=192.168.33.11
MESOS_IP=192.168.33.11
MESOS_MASTER=zk://192.168.33.10/mesos <http://192.168.33.10/mesos>
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
On all machines I have the following environment variables:
LD_LIBRARY_PATH=/path/to/mesos/lib
JAVA_HOME is also set up correctly
I then attempted to run mesos-hdfs from the master. The scheduler
launches properly.
However, when the scheduler tries to launch a task (such as a
journal node), it fails and I can see this in the stderr of the
sandbox:
mesos-containerizer: error while loading shared libraries:
libmesos-0.24.0.so <http://libmesos-0.24.0.so>: cannot open shared
object file: No such file or directory
What causes this problem and how can I resolve it?
--
Best Regards,
Haosdent Huang
Re: mesos-containerizer: error while loading shared libraries: libmesos-0.24.0.so
I have that set using the environment variable:
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
However, it doesn't seem to have any effect.
On 18/09/2015 12:27 AM, haosdent wrote:
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
Building portable binaries
Is there anyway to build portable binaries for mesos? Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr and then built mesos using the following: ../configure CC=gcc-4.8 CXX=g++-4.8 LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl make make install I then compress /tmp/mesos-build/mesos into an archive and distribute it to my machines. The only problem is that the build seems to be buggy. For example, I've been experiencing a containerization issues where the executors will crash, but not output anything useful to stderr and stdout. See https://github.com/mesosphere/hdfs/issues/194 Is there a definite way to build portable binaries that I can easily copy to another machine to run?
Re: Building portable binaries
Thanks for your pointers! I will give that a try to see if I can track down the problem. Do you mind sharing a bit more about your build process? What do your ./configure flags look like? Also, do you do a `make install` and then package up the contents? Cheers! On 18/09/2015 10:06 AM, Alexander Gallego wrote: I run a custom build of mesos and it works pretty reliably, so I'll try to share what I do in the hopes that it helps you. The symptom you are seeing of exiting before anything is output to the logs has been usually an issue with the LD_LIBRARY_PATH being screwed up. I would try to ssh into the machines and run the command manually. Native code is a bit hard to debug, but this is what I do for a cluster to figure out mesos issues. 1. Recompile with -ggdb and ship dwarf symbols (if you don't want to, you'll have to symbolize the binary later) 2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your machines 3. wrap your mesos executors (if native) as well as mesos-master and mesos-slave in a bash script that you control 4. In this bash script, set the filehandle limits from the soft to the hard 5. In this bash script, set the core dump limits to unlimited When it crashes, it will be pretty straight forward to debug with the core dump, it will almost tell you the exact assembly instruction, or exception in the code. I've never had crashes with mesos after compiling with ``` -mtune=generic ``` I had something similar to my rocksdb build (https://github.com/facebook/rocksdb/issues/690) happened to my mesos build when I was first starting and compiling with -mtune=native If you are linking the libmesos.so into another process (i.e.: you wrote your own native executor) then make sure you do something similar to the core dump limits. Note: If you are running dockerized/containerized deployments you won't be able to modify the /proc/sys/* , so this will have to be done on the host computers. I've been meaning to write a blog post about this, but this is the TL;DR. Hope this helps. - Alex On Thu, Sep 17, 2015 at 7:33 PM, F21 <f21.gro...@gmail.com <mailto:f21.gro...@gmail.com>> wrote: Is there anyway to build portable binaries for mesos? Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr and then built mesos using the following: ../configure CC=gcc-4.8 CXX=g++-4.8 LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl make make install I then compress /tmp/mesos-build/mesos into an archive and distribute it to my machines. The only problem is that the build seems to be buggy. For example, I've been experiencing a containerization issues where the executors will crash, but not output anything useful to stderr and stdout. See https://github.com/mesosphere/hdfs/issues/194 Is there a definite way to build portable binaries that I can easily copy to another machine to run? -- Alexander Gallego | http://concord.io
Re: mesos-containerizer: error while loading shared libraries: libmesos-0.24.0.so
Hey haosdent, I was the one that opened the issue :)
On 18/09/2015 1:52 PM, haosdent wrote:
Hi, @F21 You problems is similar to
https://issues.apache.org/jira/browse/MESOS-3462? I would test it
tonight and give you feedback later.
On Fri, Sep 18, 2015 at 5:59 AM, F21 <f21.gro...@gmail.com
<mailto:f21.gro...@gmail.com>> wrote:
I have that set using the environment variable:
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
However, it doesn't seem to have any effect.
On 18/09/2015 12:27 AM, haosdent wrote:
MESOS_EXECUTOR_ENVIRONMENT_VARIABLES={"LD_LIBRARY_PATH":
"/path/to/mesos/lib"}
--
Best Regards,
Haosdent Huang
How does mesos determine how much memory on a node is available for offer?
I have 3 CoreOS nodes running in vagrant. Mesos is run natively (not in docker containers). There is 1 master/slave and 2 slaves. If I ssh into one of my slaves and run free -m, I see: Total: 2005 Used 1342 Free 662 Shares 273 Buffers 13 Cached 1210 In the mesos web-ui, I see that the slave has 1002 MB of memory to offer. How is this 1002 MB determined (I am running the masters and slaves with stock defaults and no customizations)? Is the 1002MB included in the used memory (1342)? If so, why is the 662MB free? It seems to be a waste and I am sure it should be able to offer another 500MB making the total 1502MB.
Re: How does mesos determine how much memory on a node is available for offer?
There seems to be some dynamicness to it. I just bumped the memory for each VM up to 2.5GB and now mesos is offering 1.5GB on it's slave. Is there some percentage value that I can set so that more memory is available to mesos? On 3/09/2015 11:23 AM, Anand Mazumdar wrote: In case you don’t specify the resources via “—resources” flag when you start your agent, it picks up the default values. (Example: --resources="cpus:4;mem:1024;disk:2”) The default value for memory is here: https://github.com/apache/mesos/blob/master/src/slave/constants.cpp#L46 -anand
No CRAM-MD5 support for libsasl2 when compiling
I am trying to build mesos 0.23. In my case, I am trying to build mesos and all of its dependencies manually, so I can build a package that contains everything. I am building cyrus sasl 2.1.26 like so: ./configure CC=gcc-4.8 CPPFLAGS=-I/usr/include/openssl --prefix=/home/user/Desktop/libsasl2 --enable-cram make make install Mesos is being build like so (modified from the instructions in the docs): ../configure CC=gcc-4.8 CXX=g++-4.8 LD_FLAGS=-L/home/user/Desktop/libsasl2/lib SASL_PATH=/home/user/Desktop/libsasl2/lib/sasl2 --prefix=/home/user/Desktop/mesos-install --with-svn=/home/user/Desktop/svn --with-apr=/home/user/Desktop/apr --with-sasl=/home/user/Desktop/libsasl2 For some reason, mesos will error because CRAM-MD5 is not found during ./configure: checking for sasl_done in -lsasl2... yes checking SASL CRAM-MD5 support... configure: error: no --- We need CRAM-MD5 support for SASL authentication. --- However, I do have CRAM-MD5 support for my compiled libsasl2. In /home/user/Desktop/libsasl2/lib/sasl2, I see libcrammd5.so, libcrammd5.so.3, libcrammd5.so.3.0.0, libcrammd5.a and libcrammd5.la If I run the pluginviewer executable that is created when building libsasl2, it shows that the CRAM-MD5 plugin is loaded. Any ideas what might be happening here?
Re: No CRAM-MD5 support for libsasl2 when compiling
Hey Jeff, I do have all headers and libraries installed in /home/user/Desktop/libsasl2 as they are compiled manually (I plan to pack the required files together with mesos into a package later on). I can see that ./configure does find the headers for libsasl2, but it can't see the plugins (which includes CRAM-MD5) inside /home/user/Desktop/libsasl2/lib/sasl2. Sasl2 does warn that as I am installing to /home/user/Desktop/libsasl2, I would need to set SASL_PATH to /home/user/Desktop/libsasl2/lib/sasl2, but setting that environment variable didn't help. Is there anyway to let ./configure see the plugins in /home/user/Desktop/libsasl2/lib/sasl2? On 17/08/2015 2:45 PM, Jeff Schroeder wrote: Yes, the same thing happened to me at first. You need both the headers (the dev package and .h files) and the actual libraries installed. On Sunday, August 16, 2015, F21 f21.gro...@gmail.com mailto:f21.gro...@gmail.com wrote: I am trying to build mesos 0.23. In my case, I am trying to build mesos and all of its dependencies manually, so I can build a package that contains everything. I am building cyrus sasl 2.1.26 like so: ./configure CC=gcc-4.8 CPPFLAGS=-I/usr/include/openssl --prefix=/home/user/Desktop/libsasl2 --enable-cram make make install Mesos is being build like so (modified from the instructions in the docs): ../configure CC=gcc-4.8 CXX=g++-4.8 LD_FLAGS=-L/home/user/Desktop/libsasl2/lib SASL_PATH=/home/user/Desktop/libsasl2/lib/sasl2 --prefix=/home/user/Desktop/mesos-install --with-svn=/home/user/Desktop/svn --with-apr=/home/user/Desktop/apr --with-sasl=/home/user/Desktop/libsasl2 For some reason, mesos will error because CRAM-MD5 is not found during ./configure: checking for sasl_done in -lsasl2... yes checking SASL CRAM-MD5 support... configure: error: no --- We need CRAM-MD5 support for SASL authentication. --- However, I do have CRAM-MD5 support for my compiled libsasl2. In /home/user/Desktop/libsasl2/lib/sasl2, I see libcrammd5.so, libcrammd5.so.3, libcrammd5.so.3.0.0, libcrammd5.a and libcrammd5.la http://libcrammd5.la If I run the pluginviewer executable that is created when building libsasl2, it shows that the CRAM-MD5 plugin is loaded. Any ideas what might be happening here? -- Text by Jeff, typos by iPhone
Re: No CRAM-MD5 support for libsasl2 when compiling
I did some poking around. Looks like configure compiles a small c program (crammd5_installed.c) and runs it to test whether sasl2 with cram-md5 support is available. The solution is to export SASL_PATH and add the location of the compiled libraries to LD_LIBRARY_PATH: ../configure CC=gcc-4.8 CXX=g++-4.8 LD_LIBRARY_PATH=/home/user/Desktop/libsasl2/lib/ SASL_PATH=/home/user/Desktop/libsasl2/lib/sasl2 --prefix=/home/user/Desktop/mesos-install --with-svn=/home/user/Desktop/svn --with-apr=/home/user/Desktop/apr --with-sasl=/home/user/Desktop/libsasl2 On 17/08/2015 3:01 PM, Jeff Schroeder wrote: Try updating your CPPFLAGS with -I and the full path to those headers perhaps? Once I had both installed this worked for me, but I was not using nonstandard paths. On Sunday, August 16, 2015, F21 f21.gro...@gmail.com mailto:f21.gro...@gmail.com wrote: Hey Jeff, I do have all headers and libraries installed in /home/user/Desktop/libsasl2 as they are compiled manually (I plan to pack the required files together with mesos into a package later on). I can see that ./configure does find the headers for libsasl2, but it can't see the plugins (which includes CRAM-MD5) inside /home/user/Desktop/libsasl2/lib/sasl2. Sasl2 does warn that as I am installing to /home/user/Desktop/libsasl2, I would need to set SASL_PATH to /home/user/Desktop/libsasl2/lib/sasl2, but setting that environment variable didn't help. Is there anyway to let ./configure see the plugins in /home/user/Desktop/libsasl2/lib/sasl2? On 17/08/2015 2:45 PM, Jeff Schroeder wrote: Yes, the same thing happened to me at first. You need both the headers (the dev package and .h files) and the actual libraries installed. On Sunday, August 16, 2015, F21 f21.gro...@gmail.com javascript:_e(%7B%7D,'cvml','f21.gro...@gmail.com'); wrote: I am trying to build mesos 0.23. In my case, I am trying to build mesos and all of its dependencies manually, so I can build a package that contains everything. I am building cyrus sasl 2.1.26 like so: ./configure CC=gcc-4.8 CPPFLAGS=-I/usr/include/openssl --prefix=/home/user/Desktop/libsasl2 --enable-cram make make install Mesos is being build like so (modified from the instructions in the docs): ../configure CC=gcc-4.8 CXX=g++-4.8 LD_FLAGS=-L/home/user/Desktop/libsasl2/lib SASL_PATH=/home/user/Desktop/libsasl2/lib/sasl2 --prefix=/home/user/Desktop/mesos-install --with-svn=/home/user/Desktop/svn --with-apr=/home/user/Desktop/apr --with-sasl=/home/user/Desktop/libsasl2 For some reason, mesos will error because CRAM-MD5 is not found during ./configure: checking for sasl_done in -lsasl2... yes checking SASL CRAM-MD5 support... configure: error: no --- We need CRAM-MD5 support for SASL authentication. --- However, I do have CRAM-MD5 support for my compiled libsasl2. In /home/user/Desktop/libsasl2/lib/sasl2, I see libcrammd5.so, libcrammd5.so.3, libcrammd5.so.3.0.0, libcrammd5.a and libcrammd5.la http://libcrammd5.la If I run the pluginviewer executable that is created when building libsasl2, it shows that the CRAM-MD5 plugin is loaded. Any ideas what might be happening here? -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone
