Hi Gourav, Thank you for the nice informative summaries, posts like these are always educational. Keep’em coming.
Suresh > On Oct 7, 2016, at 10:56 PM, Shenoy, Gourav Ganesh <[email protected]> > wrote: > > Hi dev, > > I have been exploring different frameworks for Mesos which would help our > use-case of providing Airavata the capability to run jobs in a Mesos based > ecosystem. In particular, I have been playing around with Marathon & Chronos > and I am now going to be working on Apache Aurora. > > I have summarized my understanding about Mesos, Marathon & Chronos below. I > will send out a separate email about Aurora later. > > Apache Mesos: > > · Apache Mesos is an open-source cluster manager, in the sense that > it helps deploy & manage different frameworks (or applications) in a large > clustered environment easily. > · Mesos provides the ability to utilize underlying shared pool of > nodes as a single compute unit – That is, it can run many applications on > these nodes efficiently. > · Mesos uses the concept of “offers” for scheduling and running jobs > on the underlying nodes. When a framework (application) wants to run > computations/jobs on the cluster, Mesos will decide how many resources it > will “offer” that framework based on the availability. The framework will > then decide which resources to use from the offer, and subsequently run the > computation/job on that resource. > · In a typical cluster, you will have 3 or more Mesos masters & > multiple Mesos slaves. Multiple mesos masters help in providing high > availability – if one master goes down, Mesos will reelect a new leader > (master) – using Zookeeper. > · The task mentioned above of providing “offers” to frameworks is > done by a master, whereas the slaves are the ones who run these computations. > > · Some additional points: > o I built a Mesos cluster with 3 masters & 2 slaves on EC2. > o Each master & slave have 1GB of RAM & 1vCPU with 20GB of disk space. > > Marathon: > > · Marathon is considered a framework that runs on top of Mesos. It is > a container orchestration platform for Mesos and essentially acts as a > service scheduler. > · It is named “marathon” because it is intended for long running > applications. That is, Marathon makes sure that the service it is running > never stops – if a service goes down or the slave on which the service is run > dies, marathon keeps re-starting it on different slaves. > · In some sense Marathon is very good for ensuring high availability > of services. That is, instead of running services directly on Mesos, run it > in Marathon if you never want it to die. > Note: You can decide to run a service on multiple slave nodes and if > resources on these slaves are available, Mesos will “offer” them to Marathon. > · It is called a container orchestration platform because it > “launches” these services inside a container – either Docker OR Mesos > container. > · In my opinion it is not a suitable “job scheduler” for Airavata > because in Airavata we need to run a job and get the output rather than > keeping it running always. Instead, we can run other schedulers – > chronos/aurora as a service in Marathon. > > Chronos: > > · Chronos is a Cron scheduler for Mesos. It is good for running > scheduled jobs – jobs that need to be run for a certain number of times, > repeatedly after certain intervals. > · Chronos also provides the ability to add dependencies between jobs > – That is, if a job1 is dependent on another job2 then it will run job1 first > and then run job2 after job1 completes. It also builds a Directed Acyclic > Graph (DAG) based on these dependencies. > · Similar to Marathon, Chronos receives “offers” from Mesos master > whenever it needs to run a job on Mesos. > · Again, I found that Chronos does not fit the Airavata use-case > since I could not find a way to run one-off jobs via Chronos – you need to > specify interval time for Chronos, & Chronos then re-runs the job after that > interval is complete (even if you decide to specify num. of repetitions=1). > > > Some additional points: > · Marathon & Chronos both have REST API support – eg: you can submit > jobs via APIs along with other interactions such as list jobs, etc. > · I installed Marathon & Chronos frameworks on the Mesos master > nodes. This is how their health looks like on the Mesos dashboard: > > <image002.png> > As you can see, there are 3 active tasks running in Chronos & > 4 active tasks (long running) in Marathon. > > · I also installed Chronos as a service inside Marathon, and this is > how it looks like in the Marathon UI: > > <image004.png> > Interestingly, Chronos (as a service in Marathon) was smart enough to > identify the jobs submitted via Chronos (as a framework on Mesos) & > vice-versa. > > · Also, Mesos dashboard lists the active tasks it is running & > details about which slave the task is running on. It also lists Completed > tasks. The “Sandbox” gives you access to the stdout/stderr files for the > tasks as well as any other directories that were created as part of the task. > > <image005.png> > > Pardon me for this long email. Next, I will explore Apache Aurora which seems > a better fit for Airavata use-case because it provides the features that > Chronos supports, as well as can run one-off jobs. > > Thanks and Regards, > Gourav Shenoy > > From: "Shenoy, Gourav Ganesh" <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, September 23, 2016 at 4:43 PM > To: "[email protected]" <[email protected]> > Subject: Mesos based meta-scheduling for Airavata > > Hi Dev, > > I am working on this project of building a Mesos based meta-scheduler for > Airavata, along with Shameera & Mangirish. Here is the jira > link:https://issues.apache.org/jira/browse/AIRAVATA-2082 > <https://issues.apache.org/jira/browse/AIRAVATA-2082>. > > · We have identified some tasks that would be needed for achieving > this, and at the higher level it would consist of: > 1. Resource provisioning – We need to provision resources on cloud & hpc > infrastructures such as EC2, Jetstream, Comet, etc. > 2. Building a cluster – Deploying a Mesos cluster on set of nodes > obtained from (1) above for task management. > 3. Selecting a scheduler – We need to investigate the scheduler to use > with Mesos cluster. Some of the options are Marathon, Aurora. But we need to > find one that suits our needs of running serial as well as parallel (MPI) > jobs. > 4. Installing & running applications on this cluster – Once the cluster > has been deployed and a scheduler choice made, we need to be able to install > and run applications on this cluster using Airavata. > > · Until now we were able to look into the following: > o Resource provisioning: > § We explored several options of provisioning resources – using cloud > libraries as well as via ansible scripts. > § We built a OpenStack4J Java module which would provision instances on > OpenStack based clouds (eg: Jetstream). > § We also built a CloudBridge Python module for provisioning EC2 instances > on Amazon. CloudBridge can also be used to provision instances on OpenStack > § We wrote Ansible scripts for bringing up instances on both AWS and > OpenStack based clouds. > > § Key Points: CloudBridge, OpenStack4J are powerful libraries for resource > provisioning, but currently they do single-instance provisioning, and not > support templated boot options such as CloudFormation (for AWS) & Heat (for > OpenStack). > > o Building a cluster: > § We wrote Ansible script for deploying a Mesos-Marathon cluster on a set of > nodes. This script will install necessary dependencies such as Zookeeper. > § We tested this on OpenStack based clouds & on EC2. > § OpenStack Magnum provides excellent support for doing resource > provisioning & deploying mesos cluster, but we are running into some problems > while trying it. > > o Installing a scheduler: > § Our Ansible script is currently installing Marathon as the scheduler on > Mesos. We haven’t yet submitted jobs using Marathon. > > · Although not finalized, but we are inclined towards using Ansible > approach for the above, as Ansible also provides Python APIs and which will > allow us to integrate it with Airavata via Thrift. Hence we will be able to > easily invoke the Ansible scripts from code without needing to use the > command-line interface. > > · We are also progressively working on some work-items such as: > o Exploring options to provision and deploy a Mesos-Marathon cluster on HPC > systems such as Comet. The challenge would be to use Ansible to provision > resources and deploy the cluster. Once we have a cluster, we can try running > applications. > o Exploring different scheduler options for running serial and parallel > (MPI) jobs on such heterogeneous clusters. > o Exploring orchestration options such as OpenStack Heat, AWS > CloudFormation, OpenStack Magnum, etc. > > Any suggestions and comments are highly appreciated. > > Thanks and Regards, > Gourav Shenoy
