Re: preparing a host on task launch event
Michael, check out https://issues.apache.org/jira/browse/MESOS-2060 for a recent feature to provide task launch hooks like you're asking about, although it acts as a master/slave-specific library rather than a task-specific prep step, so you'll have to customize the behavior based on some information about the task. Alternatively, you could use the upcoming Persistent Volumes feature ( https://issues.apache.org/jira/browse/MESOS-1554) in such a way that you first launch a task to prep the state in a volume, and after its completion launch the long-running docker task that uses that volume. On Thu, Feb 19, 2015 at 6:45 PM, Michael Neale michael.ne...@gmail.com wrote: (in a vain help to try and clarify) - I started with a similar pattern to what I have seen with redis - people ensure there is a redis on each host listening on a known port so apps can use it (by setting a unique constraint on host name, and then making sure number of instances == size of cluster). I started doing the same thing with a service that provides the volume data - this works great - but has to prepare the data *before* the docker container launches - or perhaps just as it is launching (docker can't see host mounts in bind mounts after it has launched - for boring reasons...). On Fri Feb 20 2015 at 1:38:20 PM Michael Neale michael.ne...@gmail.com wrote: well not specifically talking about the mesos containerizer - it was just something I tried. The main aim is to deploy containers that can be bind mounted in a volume which is prepared on the host - the container apps (docker apps) being deployed don't particularly care how that was prepared - just that it was there. I was hoping for another task (or something) that had run before had prepared it (in some cases it may simply be rsyncing some data in place, in others, mounting a device - result is the same - a volume/path can be provided to the docker container). Does that make a little more sense ? (a bit hard to explain). On Fri Feb 20 2015 at 1:23:46 PM Tim Chen t...@mesosphere.io wrote: Hi Michael, Can you elaborate how you use the Mesos containerizer to you prepare your host? In general hooks are exactly for this purpose, which is underway right now for defining the hooks in Mesos and also allowing it to be customized. Tim On Thu, Feb 19, 2015 at 6:18 PM, Michael Neale michael.ne...@gmail.com wrote: I am currently using marathon and have a need to prepare the host in some cases (currently looking at mounting a volume that the task may need - how that device is created is out of band BTW). In theory this would be ideally done on some hook - but I am not sure where (the hook would be called before the task proper is launched) - it could be simply as part of a task launch script if a plain command. With the docker containerizer - I can actually use priv mode and control the host (if I want) - but then I would like to have this task run separately to the main marathon long running task (as it has extra access which normally apps don't need) - I could bind mount in the docker socket and launch a non priv container from within the mesos launched start container ... I can also use the default (?) mesos containerizer - which seems to let me run docker commands (ie bypassing the firstclass support in mesos for docker) but this feels like I am doing it wrong - is that wrong? So in summary: is there a concept of a pre-launch step, and should I be working around the docker containerizer by using the mesos default containerizer instead? pointers appreciated.
Re: Error when creating a new HA cluster on EC2
On 02/22/2015 06:35 AM, i...@roybos.nl wrote: Last friday i put some Ansible scripts on github for provisoning a multi AZ cluster on AWS. You could have a look at it https://github.com/roybos/aws-mesos-marathon-cluster and maybe it helps you. It basically creates a VPC within an AWS region and setups 1 node in 3 AZ's. All nodes are currently equal(mesos master and slave are on the same machine) which is fine for smaller clusters for let's say 3 to 30 nodes. Roy Bos Very cool. I'm new to ansible, but I like what I've learns so far. What would be cool is if some thing like this ansible example was for just 3 (generic linux ) system that are installed from sources. No binaries, no distro specific packages, just pure, raw sources. That way it would provide a baseline for installation on all sorts of linux (embedded, vm, container, uncommon_distro, different architectures like arm64 etc etc) based systems. Any ansible guides for generic mesos installs from sources would be of keen interests to many folks. After that in-house from sources methodology is viable, I'm quite certain companies will want to augment their local (in-house) cluster resources, with resources from vendors, particularly in a dynamic mode of utilization. Therefore, within the in-house resources mixed (supplemented) with vendor resources is the future of cluster computing. ymmv. Sadly, I see little progress on this open systems approach, and that concerns me greatly for the viability of the mesos cluster. Is it indeed going to be limited to large corporations trying to sell something nobody wants; or are there going to be open source methodologies developed that streamline the installation of mesos inside of a medium size company with modest resources. Ansible is the key technology here combined with building up a mesos cluster FROM 100% SOURCES. Put that in your presentation at mesos con(artists?) James
Re: preparing a host on task launch event
Hi Adam - yes the hooks one possibly fit the bill - not entirely clear how to use it yet. The peristent one *should* work, but the problem for me isn't so much the management of volume, but the preparation of it (I have no need to make tasks sticky when the data can freely move around anyway). On Sun Feb 22 2015 at 8:04:58 PM Adam Bordelon a...@mesosphere.io wrote: Michael, check out https://issues.apache.org/jira/browse/MESOS-2060 for a recent feature to provide task launch hooks like you're asking about, although it acts as a master/slave-specific library rather than a task-specific prep step, so you'll have to customize the behavior based on some information about the task. Alternatively, you could use the upcoming Persistent Volumes feature ( https://issues.apache.org/jira/browse/MESOS-1554) in such a way that you first launch a task to prep the state in a volume, and after its completion launch the long-running docker task that uses that volume. On Thu, Feb 19, 2015 at 6:45 PM, Michael Neale michael.ne...@gmail.com wrote: (in a vain help to try and clarify) - I started with a similar pattern to what I have seen with redis - people ensure there is a redis on each host listening on a known port so apps can use it (by setting a unique constraint on host name, and then making sure number of instances == size of cluster). I started doing the same thing with a service that provides the volume data - this works great - but has to prepare the data *before* the docker container launches - or perhaps just as it is launching (docker can't see host mounts in bind mounts after it has launched - for boring reasons...). On Fri Feb 20 2015 at 1:38:20 PM Michael Neale michael.ne...@gmail.com wrote: well not specifically talking about the mesos containerizer - it was just something I tried. The main aim is to deploy containers that can be bind mounted in a volume which is prepared on the host - the container apps (docker apps) being deployed don't particularly care how that was prepared - just that it was there. I was hoping for another task (or something) that had run before had prepared it (in some cases it may simply be rsyncing some data in place, in others, mounting a device - result is the same - a volume/path can be provided to the docker container). Does that make a little more sense ? (a bit hard to explain). On Fri Feb 20 2015 at 1:23:46 PM Tim Chen t...@mesosphere.io wrote: Hi Michael, Can you elaborate how you use the Mesos containerizer to you prepare your host? In general hooks are exactly for this purpose, which is underway right now for defining the hooks in Mesos and also allowing it to be customized. Tim On Thu, Feb 19, 2015 at 6:18 PM, Michael Neale michael.ne...@gmail.com wrote: I am currently using marathon and have a need to prepare the host in some cases (currently looking at mounting a volume that the task may need - how that device is created is out of band BTW). In theory this would be ideally done on some hook - but I am not sure where (the hook would be called before the task proper is launched) - it could be simply as part of a task launch script if a plain command. With the docker containerizer - I can actually use priv mode and control the host (if I want) - but then I would like to have this task run separately to the main marathon long running task (as it has extra access which normally apps don't need) - I could bind mount in the docker socket and launch a non priv container from within the mesos launched start container ... I can also use the default (?) mesos containerizer - which seems to let me run docker commands (ie bypassing the firstclass support in mesos for docker) but this feels like I am doing it wrong - is that wrong? So in summary: is there a concept of a pre-launch step, and should I be working around the docker containerizer by using the mesos default containerizer instead? pointers appreciated.
