Hi Marlon, "we more typically use Airavata to run scientific applications on clusters" - Our plan is to run dockerized scientific application through Airavata. Definitely our target is not to run a Tomcat webserver, I just said that as an example.
Give me some time, after I discuss with Prof. Madhu and Renan I will try to reply your queries in details. Thanks Pankaj On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <[email protected]> wrote: > Hi Pankaj— > > I may not have the full background on what you want to do, but we more > typically use Airavata to run scientific applications on clusters. This > goes through a scheduling and queuing system (Torque/MOAB or SLURM). If > you want to do this, you would first need to say how you would submit a > dockerized application using Mesos as your queueing system. You could do > this for a Tomcat server, but that is a little different from the jobs we > normally run. > > We don’t use Airavata to bootstrap itself, although that may be an > interesting way to think of things. There it is interesting to use Mesos + > Marathon to run Airavata services (which may themselves by dockerized) but > that would be done today outside Airavata. > > Marlon > > > From: Suresh Marru <[email protected]> > Date: Wednesday, October 28, 2015 at 12:25 PM > To: Airavata Dev <[email protected]> > Cc: Marlon Pierce <[email protected]> > > Subject: Re: launching a job through Airavata to Mesos cluster > > Hi Pankaj, > > You cannot do this out of the box now, but with some guidance you can > easily write a task to run Dockerized application on a Mesos cluster, > please use 0.16 for this. As Shameera explained this is a improvised > architecture and any contributions you will make will carry forward. Note: > This will require you to understand under the hood details of Airavata > Architecture but I am sure you will motivate others to work with you to > speed it up. > > Suresh > > On Oct 28, 2015, at 12:22 PM, Pankaj Saha <[email protected]> wrote: > > “launch a dockerized job” means submitting a docker container for any > specific job (say a java webserver docker container) through Airavata. > What I remember Airavata php gateway can not submit any docker > container(correct me if I am wrong). What I want to figure out initially is > whether its feasible to submit a docker container with help of Airavata > into a Mesos cluster. > > Thanks > Pankaj > > On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <[email protected]> wrote: > >> I’ll add: if submitting a job to (for example) a SLURM queuing system, we >> need to create the correct SLURM submission script and submit it by >> executing the correct command line operation (sbatch). >> >> From: Marlon Pierce <[email protected]> >> Reply-To: Airavata Dev <[email protected]> >> Date: Wednesday, October 28, 2015 at 12:08 PM >> To: Airavata Dev <[email protected]>, Suresh Marru < >> [email protected]>, Pankaj Saha <[email protected]> >> >> Subject: Re: launching a job through Airavata to Mesos cluster >> >> Hi Pankaj, can you say more about what you mean by “launch a dockerized >> job”? >> >> Marlon >> >> >> From: Pankaj Saha <[email protected]> >> Reply-To: Airavata Dev <[email protected]> >> Date: Wednesday, October 28, 2015 at 11:56 AM >> To: Suresh Marru <[email protected]> >> Cc: Airavata Dev <[email protected]> >> Subject: Re: launching a job through Airavata to Mesos cluster >> >> Hi Suresh, >> >> My initial understanding is, I have to launch a dockerized job through >> Airavata which will be run in the Mesos cluster. I was looking for the >> code which submits jobs and wanted to make changes such a way that it can >> submit docker containers to Mesos/Marathon cluster. >> >> I can use 0.15 branch and I have no idea about data transfer protocol and >> job submission protocols that Shameera has mentioned. I may want to submit >> jobs by submitting a JSON through command line or any other way that you >> guys feel is more appropriate. >> >> I can talk to Prof. Madhu and let you know more on the requirement. >> >> Thanks >> Pankaj >> >> >> >> >> >> >> >> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <[email protected]> wrote: >> >>> Pankaj can you clarify the following: >>> >>> Do you want an Airavata instance to run some dockerized applications >>> scheduled by Mesos? Or do you just need a client which will connect to >>> Airavata hosted and managed by Mesos/Marathon? >>> >>> Suresh >>> >>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka < >>> [email protected]> wrote: >>> >>> Hi Pankaj, >>> >>> Wich version of Airavata you are working on? what is the data transfer >>> protocol? What is the job submission protocol? >>> >>> Short answer: if you are using Airavata 0.15 then you need to write new >>> Provider implementation to submit the request to Mesos/Marathon cluster. >>> But if you are using Airavata 16.0 which is current master, then you need >>> to write JobSubmissionTask implementation. Either case you can go through >>> the existing implementations, for Provider implementation see SSHProvider >>> and JobSubmissionTask implementation see SSHJobSubmissionTask. >>> >>> If I get the answers to my questions then i can provide exactly what you >>> need to do. BTW we have cleaned our internal architecture in Airavata 16.0, >>> as a developer you would find it easy to work with Airavata 16.0 that >>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0. >>> >>> Regards, >>> Shameera. >>> >>> >>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <[email protected]> >>> wrote: >>> >>>> Hello Shameera, >>>> I am working on jet-stream project, where I have to find out a way to >>>> submit a job in mesos/marathon cluster through Airavata client. I don't >>>> have much idea from where to start looking into. Can you please give some >>>> clue so that I can start working and making changes to java code for the >>>> same. >>>> >>>> Thanks >>>> Pankaj >>>> >>>> -- >>> Shameera Rathnayaka >>> >>> >>> >> > >
