As Suresh said I think this is the time when I have to understand under the hood details of Airavata Architecture, which I could not gain much during GSoC.
On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpi...@iu.edu> wrote: > The “hello world” for us is usually to just run some linux command on the > remote queuing system. So that is something you could easily wrap via > docker and submit via Mesos (I hope it is easy…). Related to this is > determining that the job has gone through various states (queued, > executing, completed) and communicating that back to Airavata. > > > Marlon > > > From: Pankaj Saha <psa...@binghamton.edu> > Date: Wednesday, October 28, 2015 at 12:54 PM > To: Marlon Pierce <marpi...@iu.edu> > Cc: Suresh Marru <sma...@apache.org>, Airavata Dev < > dev@airavata.apache.org> > > Subject: Re: launching a job through Airavata to Mesos cluster > > Hi Marlon, > "we more typically use Airavata to run scientific applications on > clusters" - Our plan is to run dockerized scientific application through A > iravata. > Definitely our target is not to run a Tomcat webserver, I just said that > as an example. > > Give me some time, after I discuss with Prof. Madhu and Renan I will try > to reply your queries in details. > > Thanks > Pankaj > > On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu> wrote: > >> Hi Pankaj— >> >> I may not have the full background on what you want to do, but we more >> typically use Airavata to run scientific applications on clusters. This >> goes through a scheduling and queuing system (Torque/MOAB or SLURM). If >> you want to do this, you would first need to say how you would submit a >> dockerized application using Mesos as your queueing system. You could do >> this for a Tomcat server, but that is a little different from the jobs we >> normally run. >> >> We don’t use Airavata to bootstrap itself, although that may be an >> interesting way to think of things. There it is interesting to use Mesos + >> Marathon to run Airavata services (which may themselves by dockerized) but >> that would be done today outside Airavata. >> >> Marlon >> >> >> From: Suresh Marru <sma...@apache.org> >> Date: Wednesday, October 28, 2015 at 12:25 PM >> To: Airavata Dev <dev@airavata.apache.org> >> Cc: Marlon Pierce <marpi...@iu.edu> >> >> Subject: Re: launching a job through Airavata to Mesos cluster >> >> Hi Pankaj, >> >> You cannot do this out of the box now, but with some guidance you can >> easily write a task to run Dockerized application on a Mesos cluster, >> please use 0.16 for this. As Shameera explained this is a improvised >> architecture and any contributions you will make will carry forward. Note: >> This will require you to understand under the hood details of Airavata >> Architecture but I am sure you will motivate others to work with you to >> speed it up. >> >> Suresh >> >> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote: >> >> “launch a dockerized job” means submitting a docker container for any >> specific job (say a java webserver docker container) through Airavata. >> What I remember Airavata php gateway can not submit any docker >> container(correct me if I am wrong). What I want to figure out initially is >> whether its feasible to submit a docker container with help of Airavata >> into a Mesos cluster. >> >> Thanks >> Pankaj >> >> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpi...@iu.edu> wrote: >> >>> I’ll add: if submitting a job to (for example) a SLURM queuing system, >>> we need to create the correct SLURM submission script and submit it by >>> executing the correct command line operation (sbatch). >>> >>> From: Marlon Pierce <marpi...@iu.edu> >>> Reply-To: Airavata Dev <dev@airavata.apache.org> >>> Date: Wednesday, October 28, 2015 at 12:08 PM >>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru < >>> sma...@apache.org>, Pankaj Saha <psa...@binghamton.edu> >>> >>> Subject: Re: launching a job through Airavata to Mesos cluster >>> >>> Hi Pankaj, can you say more about what you mean by “launch a dockerized >>> job”? >>> >>> Marlon >>> >>> >>> From: Pankaj Saha <psa...@binghamton.edu> >>> Reply-To: Airavata Dev <dev@airavata.apache.org> >>> Date: Wednesday, October 28, 2015 at 11:56 AM >>> To: Suresh Marru <sma...@apache.org> >>> Cc: Airavata Dev <dev@airavata.apache.org> >>> Subject: Re: launching a job through Airavata to Mesos cluster >>> >>> Hi Suresh, >>> >>> My initial understanding is, I have to launch a dockerized job through >>> Airavata which will be run in the Mesos cluster. I was looking for the >>> code which submits jobs and wanted to make changes such a way that it can >>> submit docker containers to Mesos/Marathon cluster. >>> >>> I can use 0.15 branch and I have no idea about data transfer protocol >>> and job submission protocols that Shameera has mentioned. I may want to >>> submit jobs by submitting a JSON through command line or any other way that >>> you guys feel is more appropriate. >>> >>> I can talk to Prof. Madhu and let you know more on the requirement. >>> >>> Thanks >>> Pankaj >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sma...@apache.org> >>> wrote: >>> >>>> Pankaj can you clarify the following: >>>> >>>> Do you want an Airavata instance to run some dockerized applications >>>> scheduled by Mesos? Or do you just need a client which will connect to >>>> Airavata hosted and managed by Mesos/Marathon? >>>> >>>> Suresh >>>> >>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka < >>>> shameerai...@gmail.com> wrote: >>>> >>>> Hi Pankaj, >>>> >>>> Wich version of Airavata you are working on? what is the data transfer >>>> protocol? What is the job submission protocol? >>>> >>>> Short answer: if you are using Airavata 0.15 then you need to write >>>> new Provider implementation to submit the request to Mesos/Marathon >>>> cluster. But if you are using Airavata 16.0 which is current master, then >>>> you need to write JobSubmissionTask implementation. Either case you can go >>>> through the existing implementations, for Provider implementation see >>>> SSHProvider and JobSubmissionTask implementation see >>>> SSHJobSubmissionTask. >>>> >>>> If I get the answers to my questions then i can provide exactly what >>>> you need to do. BTW we have cleaned our internal architecture in Airavata >>>> 16.0, as a developer you would find it easy to work with Airavata 16.0 that >>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0. >>>> >>>> Regards, >>>> Shameera. >>>> >>>> >>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psa...@binghamton.edu> >>>> wrote: >>>> >>>>> Hello Shameera, >>>>> I am working on jet-stream project, where I have to find out a way to >>>>> submit a job in mesos/marathon cluster through Airavata client. I don't >>>>> have much idea from where to start looking into. Can you please give some >>>>> clue so that I can start working and making changes to java code for the >>>>> same. >>>>> >>>>> Thanks >>>>> Pankaj >>>>> >>>>> -- >>>> Shameera Rathnayaka >>>> >>>> >>>> >>> >> >> >
