The “hello world” for us is usually to just run some linux command on the remote queuing system. So that is something you could easily wrap via docker and submit via Mesos (I hope it is easy…). Related to this is determining that the job has gone through various states (queued, executing, completed) and communicating that back to Airavata.
Marlon From: Pankaj Saha <[email protected]<mailto:[email protected]>> Date: Wednesday, October 28, 2015 at 12:54 PM To: Marlon Pierce <[email protected]<mailto:[email protected]>> Cc: Suresh Marru <[email protected]<mailto:[email protected]>>, Airavata Dev <[email protected]<mailto:[email protected]>> Subject: Re: launching a job through Airavata to Mesos cluster Hi Marlon, "we more typically use Airavata to run scientific applications on clusters" - Our plan is to run dockerized scientific application through Airavata. Definitely our target is not to run a Tomcat webserver, I just said that as an example. Give me some time, after I discuss with Prof. Madhu and Renan I will try to reply your queries in details. Thanks Pankaj On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <[email protected]<mailto:[email protected]>> wrote: Hi Pankaj— I may not have the full background on what you want to do, but we more typically use Airavata to run scientific applications on clusters. This goes through a scheduling and queuing system (Torque/MOAB or SLURM). If you want to do this, you would first need to say how you would submit a dockerized application using Mesos as your queueing system. You could do this for a Tomcat server, but that is a little different from the jobs we normally run. We don’t use Airavata to bootstrap itself, although that may be an interesting way to think of things. There it is interesting to use Mesos + Marathon to run Airavata services (which may themselves by dockerized) but that would be done today outside Airavata. Marlon From: Suresh Marru <[email protected]<mailto:[email protected]>> Date: Wednesday, October 28, 2015 at 12:25 PM To: Airavata Dev <[email protected]<mailto:[email protected]>> Cc: Marlon Pierce <[email protected]<mailto:[email protected]>> Subject: Re: launching a job through Airavata to Mesos cluster Hi Pankaj, You cannot do this out of the box now, but with some guidance you can easily write a task to run Dockerized application on a Mesos cluster, please use 0.16 for this. As Shameera explained this is a improvised architecture and any contributions you will make will carry forward. Note: This will require you to understand under the hood details of Airavata Architecture but I am sure you will motivate others to work with you to speed it up. Suresh On Oct 28, 2015, at 12:22 PM, Pankaj Saha <[email protected]<mailto:[email protected]>> wrote: “launch a dockerized job” means submitting a docker container for any specific job (say a java webserver docker container) through Airavata. What I remember Airavata php gateway can not submit any docker container(correct me if I am wrong). What I want to figure out initially is whether its feasible to submit a docker container with help of Airavata into a Mesos cluster. Thanks Pankaj On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <[email protected]<mailto:[email protected]>> wrote: I’ll add: if submitting a job to (for example) a SLURM queuing system, we need to create the correct SLURM submission script and submit it by executing the correct command line operation (sbatch). From: Marlon Pierce <[email protected]<mailto:[email protected]>> Reply-To: Airavata Dev <[email protected]<mailto:[email protected]>> Date: Wednesday, October 28, 2015 at 12:08 PM To: Airavata Dev <[email protected]<mailto:[email protected]>>, Suresh Marru <[email protected]<mailto:[email protected]>>, Pankaj Saha <[email protected]<mailto:[email protected]>> Subject: Re: launching a job through Airavata to Mesos cluster Hi Pankaj, can you say more about what you mean by “launch a dockerized job”? Marlon From: Pankaj Saha <[email protected]<mailto:[email protected]>> Reply-To: Airavata Dev <[email protected]<mailto:[email protected]>> Date: Wednesday, October 28, 2015 at 11:56 AM To: Suresh Marru <[email protected]<mailto:[email protected]>> Cc: Airavata Dev <[email protected]<mailto:[email protected]>> Subject: Re: launching a job through Airavata to Mesos cluster Hi Suresh, My initial understanding is, I have to launch a dockerized job through Airavata which will be run in the Mesos cluster. I was looking for the code which submits jobs and wanted to make changes such a way that it can submit docker containers to Mesos/Marathon cluster. I can use 0.15 branch and I have no idea about data transfer protocol and job submission protocols that Shameera has mentioned. I may want to submit jobs by submitting a JSON through command line or any other way that you guys feel is more appropriate. I can talk to Prof. Madhu and let you know more on the requirement. Thanks Pankaj On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <[email protected]<mailto:[email protected]>> wrote: Pankaj can you clarify the following: Do you want an Airavata instance to run some dockerized applications scheduled by Mesos? Or do you just need a client which will connect to Airavata hosted and managed by Mesos/Marathon? Suresh On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <[email protected]<mailto:[email protected]>> wrote: Hi Pankaj, Wich version of Airavata you are working on? what is the data transfer protocol? What is the job submission protocol? Short answer: if you are using Airavata 0.15 then you need to write new Provider implementation to submit the request to Mesos/Marathon cluster. But if you are using Airavata 16.0 which is current master, then you need to write JobSubmissionTask implementation. Either case you can go through the existing implementations, for Provider implementation see SSHProvider and JobSubmissionTask implementation see SSHJobSubmissionTask. If I get the answers to my questions then i can provide exactly what you need to do. BTW we have cleaned our internal architecture in Airavata 16.0, as a developer you would find it easy to work with Airavata 16.0 that Airavata 15.0. But notice master is not yet stable as Airavata 15.0. Regards, Shameera. On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <[email protected]<mailto:[email protected]>> wrote: Hello Shameera, I am working on jet-stream project, where I have to find out a way to submit a job in mesos/marathon cluster through Airavata client. I don't have much idea from where to start looking into. Can you please give some clue so that I can start working and making changes to java code for the same. Thanks Pankaj -- Shameera Rathnayaka
