System dependencies with Mesos
I’m investigating the possibility of using Mesos to solve the problem of resource allocation between a Hadoop cluster and set of Jenkins slaves (and I like the possibility of being able to easily deploy other frameworks). One of the biggest overhanging questions I can’t seem to find an answer to is how to manage system dependencies across a wide variety of frameworks, and jobs running within those frameworks. I came across this thread (http://www.mail-archive.com/user@mesos.apache.org/msg00301.html) and caching executor files seems to be the running solution, though not implemented yet. I too would really like to avoid shipping system dependencies (c-deps for python packages, as an example) along with every single job, and i’m especially unsure how this would interact with the Hadoop/Jenkins mesos schedulers (as each hadoop job may require it’s own system dependencies). More importantly, the architecture of the machine submitting the job is often different from the slaves so we can’t simply ship all the built dependencies with the task. We’re solving this problem at the moment for Hadoop by installing all dependencies we require on every hadoop task tracker node, which is far from ideal. For jenkins, we’re using Docker to isolate execution of different types of jobs, and built all system dependencies for a suite of jobs into docker images. I like the idea of continuing down the path of Docker for process isolation and system dependency management, but I don’t see any easy way for this to interact with the existing hadoop/jenkins/etc. schedulers. I guess it’d require us to build our own schedulers/executors that wrapped the process in a Docker container. I’d love to hear how others are solving this problem… and/or whether Docker seems like the wrong way to go. — Tom Arnfeld Developer // DueDil
Re: [VOTE] Release Apache Mesos 0.16.0 (rc5)
Looks like we the 'm4' directory in stout was not added to the distribution: $ ./bootstrap ... m4/acx_pthread.m4:363: ACX_PTHREAD is expanded from... configure.ac:87: the top level autoreconf: configure.ac: adding subdirectory 3rdparty/stout to autoreconf autoreconf: Entering directory `3rdparty/stout' aclocal: error: couldn't open directory 'm4': No such file or directory autoreconf: aclocal failed with exit status: 1 On Fri, Jan 31, 2014 at 11:26 AM, Vinod Kone vinodk...@gmail.com wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.16.0. 0.16.0 includes the following: * The primary feature of this release is major refactoring work on the master election and detection process to improve its reliability and flexibility. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.16.0-rc5 The candidate for Mesos 0.16.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.16.0-rc5/mesos-0.16.0.tar.gz The tag to be voted on is 0.16.0-rc5: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.16.0-rc5 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.16.0-rc5/mesos-0.16.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.16.0-rc5/mesos-0.16.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1002 Please vote on releasing this package as Apache Mesos 0.16.0! The vote is open until Mon Feb 3 11:14:30 PST 2014 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.16.0 [ ] -1 Do not release this package because ... Thanks,
