Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Hi Suresh and Marlon
I am trying to launch a hello world sample application from my local docker
Airavata PGA. Can I get a demo admin role for few days so that I can create
an application and test some sample experiments? It looks like without
admin role I can not create applications.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpi...@iu.edu> wrote:

> But you could answer my implied question below with zero knowledge of
> Airavata.
>
> From: Pankaj Saha <psa...@binghamton.edu>
> Reply-To: Airavata Dev <dev@airavata.apache.org>
> Date: Wednesday, October 28, 2015 at 1:48 PM
>
> To: Marlon Pierce <marpi...@iu.edu>
> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
> dev@airavata.apache.org>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> As Suresh said I think this is the time  when I have to understand under
> the hood details of Airavata Architecture, which I could not gain much
> during GSoC.
>
> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>
>> The “hello world” for us is usually to just run some linux command on the
>> remote queuing system. So that is something you could easily wrap via
>> docker and submit via Mesos (I hope it is easy…).  Related to this is
>> determining that the job has gone through various states (queued,
>> executing, completed) and communicating that back to Airavata.
>>
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <psa...@binghamton.edu>
>> Date: Wednesday, October 28, 2015 at 12:54 PM
>> To: Marlon Pierce <marpi...@iu.edu>
>> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
>> dev@airavata.apache.org>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Marlon,
>> "we more typically use Airavata to run scientific applications on
>> clusters" - Our plan is to run dockerized scientific application through
>> Airavata.
>> Definitely our target is not to run a Tomcat webserver, I just said that
>> as an example.
>>
>> Give me some time, after I discuss with Prof. Madhu and Renan I will try
>> to reply your queries in details.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>>
>>> Hi Pankaj—
>>>
>>> I may not have the full background on what you want to do, but we more
>>> typically use Airavata to run scientific applications on clusters. This
>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>>> you want to do this, you would first need to say how you would submit a
>>> dockerized application using Mesos as your queueing system. You could do
>>> this for a Tomcat server, but that is a little different from the jobs we
>>> normally run.
>>>
>>> We don’t use Airavata to bootstrap itself, although that may be an
>>> interesting way to think of things.  There it is interesting to use Mesos +
>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>> that would be done today outside Airavata.
>>>
>>> Marlon
>>>
>>>
>>> From: Suresh Marru <sma...@apache.org>
>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>> To: Airavata Dev <dev@airavata.apache.org>
>>> Cc: Marlon Pierce <marpi...@iu.edu>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Pankaj,
>>>
>>> You cannot do this out of the box now, but with some guidance you can
>>> easily write a task to run  Dockerized application on a Mesos cluster,
>>> please use 0.16 for this. As Shameera explained this is a improvised
>>> architecture and any contributions you will make will carry forward. Note:
>>> This will require you to understand under the hood details of Airavata
>>> Architecture but I am sure you will motivate others to work with you to
>>> speed it up.
>>>
>>> Suresh
>>>
>>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote:
>>>
>>> “launch a dockerized job”  means submitting a docker container for any
>>> specific job (say a java webserver docker container) through Airavata.
>>> What I remember Airavata php gateway can not submit any docker
>>> container(correct me if I am wrong). What I want to figure out initially is
>>> whether its feasible to submit a docker container with help of Airavata
>>> into a Mesos cluste

Re: launching a job through Airavata to Mesos cluster

[Supun Nakandala]

Hi Pankaj,

You can set to admin role to Internal/everyone in pga_config.php (in
/app/config/pga_config.php) and you will admin permissions.

On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <psa...@binghamton.edu> wrote:

> Hi Suresh and Marlon
> I am trying to launch a hello world sample application from my local
> docker Airavata PGA. Can I get a demo admin role for few days so that I can
> create an application and test some sample experiments? It looks like
> without admin role I can not create applications.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>
>> But you could answer my implied question below with zero knowledge of
>> Airavata.
>>
>> From: Pankaj Saha <psa...@binghamton.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>
>> To: Marlon Pierce <marpi...@iu.edu>
>> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
>> dev@airavata.apache.org>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> As Suresh said I think this is the time  when I have to understand under
>> the hood details of Airavata Architecture, which I could not gain much
>> during GSoC.
>>
>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>>
>>> The “hello world” for us is usually to just run some linux command on
>>> the remote queuing system. So that is something you could easily wrap via
>>> docker and submit via Mesos (I hope it is easy…).  Related to this is
>>> determining that the job has gone through various states (queued,
>>> executing, completed) and communicating that back to Airavata.
>>>
>>>
>>> Marlon
>>>
>>>
>>> From: Pankaj Saha <psa...@binghamton.edu>
>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>> To: Marlon Pierce <marpi...@iu.edu>
>>> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
>>> dev@airavata.apache.org>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Marlon,
>>> "we more typically use Airavata to run scientific applications on
>>> clusters" - Our plan is to run dockerized scientific application
>>> through Airavata.
>>> Definitely our target is not to run a Tomcat webserver, I just said that
>>> as an example.
>>>
>>> Give me some time, after I discuss with Prof. Madhu and Renan I will try
>>> to reply your queries in details.
>>>
>>> Thanks
>>> Pankaj
>>>
>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu>
>>> wrote:
>>>
>>>> Hi Pankaj—
>>>>
>>>> I may not have the full background on what you want to do, but we more
>>>> typically use Airavata to run scientific applications on clusters. This
>>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>>>> you want to do this, you would first need to say how you would submit a
>>>> dockerized application using Mesos as your queueing system. You could do
>>>> this for a Tomcat server, but that is a little different from the jobs we
>>>> normally run.
>>>>
>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>> interesting way to think of things.  There it is interesting to use Mesos +
>>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>>> that would be done today outside Airavata.
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Suresh Marru <sma...@apache.org>
>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>> Cc: Marlon Pierce <marpi...@iu.edu>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Pankaj,
>>>>
>>>> You cannot do this out of the box now, but with some guidance you can
>>>> easily write a task to run  Dockerized application on a Mesos cluster,
>>>> please use 0.16 for this. As Shameera explained this is a improvised
>>>> architecture and any contributions you will make will carry forward. Note:
>>>> This will require you to understand under the hood details of Airavata
>>>> Architecture but I am sure you will motivate others to work with you to
>&

Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Hi Supun,
I need a small clarification here. I am using local docker for PGA and
servers but I am using the sam credential service that you guys are using.
So do you think making local changes in the pga_config will do the work?

Thanks
Pankaj

On Sat, Nov 14, 2015 at 3:01 PM, Supun Nakandala <supun.nakand...@gmail.com>
wrote:

> Hi Pankaj,
>
> You can set to admin role to Internal/everyone in pga_config.php (in
> /app/config/pga_config.php) and you will admin permissions.
>
> On Sat, Nov 14, 2015 at 2:06 PM, Pankaj Saha <psa...@binghamton.edu>
> wrote:
>
>> Hi Suresh and Marlon
>> I am trying to launch a hello world sample application from my local
>> docker Airavata PGA. Can I get a demo admin role for few days so that I can
>> create an application and test some sample experiments? It looks like
>> without admin role I can not create applications.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 1:55 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>>
>>> But you could answer my implied question below with zero knowledge of
>>> Airavata.
>>>
>>> From: Pankaj Saha <psa...@binghamton.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 1:48 PM
>>>
>>> To: Marlon Pierce <marpi...@iu.edu>
>>> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
>>> dev@airavata.apache.org>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> As Suresh said I think this is the time  when I have to understand
>>> under the hood details of Airavata Architecture, which I could not gain
>>> much during GSoC.
>>>
>>> On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>>>
>>>> The “hello world” for us is usually to just run some linux command on
>>>> the remote queuing system. So that is something you could easily wrap via
>>>> docker and submit via Mesos (I hope it is easy…).  Related to this is
>>>> determining that the job has gone through various states (queued,
>>>> executing, completed) and communicating that back to Airavata.
>>>>
>>>>
>>>> Marlon
>>>>
>>>>
>>>> From: Pankaj Saha <psa...@binghamton.edu>
>>>> Date: Wednesday, October 28, 2015 at 12:54 PM
>>>> To: Marlon Pierce <marpi...@iu.edu>
>>>> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
>>>> dev@airavata.apache.org>
>>>>
>>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>>
>>>> Hi Marlon,
>>>> "we more typically use Airavata to run scientific applications on
>>>> clusters" - Our plan is to run dockerized scientific application
>>>> through Airavata.
>>>> Definitely our target is not to run a Tomcat webserver, I just said
>>>> that as an example.
>>>>
>>>> Give me some time, after I discuss with Prof. Madhu and Renan I will
>>>> try to reply your queries in details.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu>
>>>> wrote:
>>>>
>>>>> Hi Pankaj—
>>>>>
>>>>> I may not have the full background on what you want to do, but we more
>>>>> typically use Airavata to run scientific applications on clusters. This
>>>>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>>>>> you want to do this, you would first need to say how you would submit a
>>>>> dockerized application using Mesos as your queueing system. You could do
>>>>> this for a Tomcat server, but that is a little different from the jobs we
>>>>> normally run.
>>>>>
>>>>> We don’t use Airavata to bootstrap itself, although that may be an
>>>>> interesting way to think of things.  There it is interesting to use Mesos 
>>>>> +
>>>>> Marathon to run Airavata services (which may themselves by dockerized) but
>>>>> that would be done today outside Airavata.
>>>>>
>>>>> Marlon
>>>>>
>>>>>
>>>>> From: Suresh Marru <sma...@apache.org>
>>>>> Date: Wednesday, October 28, 2015 at 12:25 PM
>>>>> To: Airavata Dev <dev@airavata.apache.org>
>>>>

Re: launching a job through Airavata to Mesos cluster

[Shameera Rathnayaka]

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer
protocol? What is the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new
Provider implementation to submit the request to Mesos/Marathon cluster.
But if you are using Airavata 16.0 which is current master, then you need
to write JobSubmissionTask implementation. Either case you can go through
the existing implementations, for Provider implementation see  SSHProvider
 and JobSubmissionTask implementation see SSHJobSubmissionTask.

If I get the answers to my questions then i can provide exactly what you
need to do. BTW we have cleaned our internal architecture in Airavata 16.0,
as a developer you would find it easy to work with Airavata 16.0 that
Airavata 15.0. But notice master is not yet stable as Airavata 15.0.

Regards,
Shameera.


On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha  wrote:

> Hello Shameera,
> I am working on jet-stream project, where I have to find out a way to
> submit a job in mesos/marathon cluster through Airavata client. I don't
> have much idea from where to start looking into. Can you please give some
> clue so that I can start working and making changes to java code for the
> same.
>
> Thanks
> Pankaj
>
> --
Shameera Rathnayaka

Re: launching a job through Airavata to Mesos cluster

[Suresh Marru]

Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled 
by Mesos? Or do you just need a client which will connect to Airavata hosted 
and managed by Mesos/Marathon? 

Suresh

> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka  
> wrote:
> 
> Hi Pankaj, 
> 
> Wich version of Airavata you are working on?  what is the data transfer 
> protocol? What is the job submission protocol? 
> 
> Short answer:  if you are using Airavata 0.15 then you need to write new 
> Provider implementation to submit the request to Mesos/Marathon cluster.  But 
> if you are using Airavata 16.0 which is current master, then you need to 
> write JobSubmissionTask implementation. Either case you can go through the 
> existing implementations, for Provider implementation see  SSHProvider  and 
> JobSubmissionTask implementation see SSHJobSubmissionTask.
> 
> If I get the answers to my questions then i can provide exactly what you need 
> to do. BTW we have cleaned our internal architecture in Airavata 16.0, as a 
> developer you would find it easy to work with Airavata 16.0 that Airavata 
> 15.0. But notice master is not yet stable as Airavata 15.0.
> 
> Regards,
> Shameera.
>  
> 
> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha  > wrote:
> Hello Shameera,
> I am working on jet-stream project, where I have to find out a way to submit 
> a job in mesos/marathon cluster through Airavata client. I don't have much 
> idea from where to start looking into. Can you please give some clue so that 
> I can start working and making changes to java code for the same.
> 
> Thanks
> Pankaj
> 
> -- 
> Shameera Rathnayaka

Re: launching a job through Airavata to Mesos cluster

[Suresh Marru]

Hi Pankaj,

You cannot do this out of the box now, but with some guidance you can easily 
write a task to run  Dockerized application on a Mesos cluster, please use 0.16 
for this. As Shameera explained this is a improvised architecture and any 
contributions you will make will carry forward. Note: This will require you to 
understand under the hood details of Airavata Architecture but I am sure you 
will motivate others to work with you to speed it up. 

Suresh

> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote:
> 
> “launch a dockerized job”  means submitting a docker container for any 
> specific job (say a java webserver docker container) through Airavata. 
> What I remember Airavata php gateway can not submit any docker 
> container(correct me if I am wrong). What I want to figure out initially is 
> whether its feasible to submit a docker container with help of Airavata into 
> a Mesos cluster.
> 
> Thanks
> Pankaj
> 
> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpi...@iu.edu 
> <mailto:marpi...@iu.edu>> wrote:
> I’ll add: if submitting a job to (for example) a SLURM queuing system, we 
> need to create the correct SLURM submission script and submit it by executing 
> the correct command line operation (sbatch).
> 
> From: Marlon Pierce <marpi...@iu.edu <mailto:marpi...@iu.edu>>
> Reply-To: Airavata Dev <dev@airavata.apache.org 
> <mailto:dev@airavata.apache.org>>
> Date: Wednesday, October 28, 2015 at 12:08 PM
> To: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>, 
> Suresh Marru <sma...@apache.org <mailto:sma...@apache.org>>, Pankaj Saha 
> <psa...@binghamton.edu <mailto:psa...@binghamton.edu>>
> 
> Subject: Re: launching a job through Airavata to Mesos cluster
> 
> Hi Pankaj, can you say more about what you mean by “launch a dockerized job”? 
>   
> 
> Marlon
> 
> 
> From: Pankaj Saha <psa...@binghamton.edu <mailto:psa...@binghamton.edu>>
> Reply-To: Airavata Dev <dev@airavata.apache.org 
> <mailto:dev@airavata.apache.org>>
> Date: Wednesday, October 28, 2015 at 11:56 AM
> To: Suresh Marru <sma...@apache.org <mailto:sma...@apache.org>>
> Cc: Airavata Dev <dev@airavata.apache.org <mailto:dev@airavata.apache.org>>
> Subject: Re: launching a job through Airavata to Mesos cluster
> 
> Hi Suresh,
> 
> My initial understanding is, I have to launch a dockerized job through 
> Airavata which will be run in the Mesos cluster.  I was looking for the code 
> which submits jobs and wanted to make changes such a way that it can submit 
> docker containers to Mesos/Marathon cluster.
> 
> I can use 0.15 branch and I have no idea about data transfer protocol and job 
> submission protocols that Shameera has mentioned. I may want to submit jobs 
> by submitting a JSON through command line or any other way that you guys feel 
> is more appropriate.
> 
> I can talk to Prof. Madhu and let you know more on the requirement.
> 
> Thanks
> Pankaj
> 
> 
> 
> 
> 
> 
> 
> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sma...@apache.org 
> <mailto:sma...@apache.org>> wrote:
> Pankaj can you clarify the following:
> 
> Do you want an Airavata instance to run some dockerized applications 
> scheduled by Mesos? Or do you just need a client which will connect to 
> Airavata hosted and managed by Mesos/Marathon? 
> 
> Suresh
> 
>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <shameerai...@gmail.com 
>> <mailto:shameerai...@gmail.com>> wrote:
>> 
>> Hi Pankaj, 
>> 
>> Wich version of Airavata you are working on?  what is the data transfer 
>> protocol? What is the job submission protocol? 
>> 
>> Short answer:  if you are using Airavata 0.15 then you need to write new 
>> Provider implementation to submit the request to Mesos/Marathon cluster.  
>> But if you are using Airavata 16.0 which is current master, then you need to 
>> write JobSubmissionTask implementation. Either case you can go through the 
>> existing implementations, for Provider implementation see  SSHProvider  and 
>> JobSubmissionTask implementation see SSHJobSubmissionTask.
>> 
>> If I get the answers to my questions then i can provide exactly what you 
>> need to do. BTW we have cleaned our internal architecture in Airavata 16.0, 
>> as a developer you would find it easy to work with Airavata 16.0 that 
>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>> 
>> Regards,
>> Shameera.
>>  
>> 
>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psa...@binghamton.edu 
>> <mailto:psa...@binghamton.edu>> wrote:
>> Hello Shameera,
>> I am working on jet-stream project, where I have to find out a way to submit 
>> a job in mesos/marathon cluster through Airavata client. I don't have much 
>> idea from where to start looking into. Can you please give some clue so that 
>> I can start working and making changes to java code for the same.
>> 
>> Thanks
>> Pankaj
>> 
>> -- 
>> Shameera Rathnayaka
> 
> 
> 

Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through
Airavata which will be run in the Mesos cluster.  I was looking for the
code which submits jobs and wanted to make changes such a way that it can
submit docker containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and
job submission protocols that Shameera has mentioned. I may want to submit
jobs by submitting a JSON through command line or any other way that you
guys feel is more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru  wrote:

> Pankaj can you clarify the following:
>
> Do you want an Airavata instance to run some dockerized applications
> scheduled by Mesos? Or do you just need a client which will connect to
> Airavata hosted and managed by Mesos/Marathon?
>
> Suresh
>
> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka 
> wrote:
>
> Hi Pankaj,
>
> Wich version of Airavata you are working on?  what is the data transfer
> protocol? What is the job submission protocol?
>
> Short answer:  if you are using Airavata 0.15 then you need to write new
> Provider implementation to submit the request to Mesos/Marathon cluster.
> But if you are using Airavata 16.0 which is current master, then you need
> to write JobSubmissionTask implementation. Either case you can go through
> the existing implementations, for Provider implementation see  SSHProvider
>  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>
> If I get the answers to my questions then i can provide exactly what you
> need to do. BTW we have cleaned our internal architecture in Airavata 16.0,
> as a developer you would find it easy to work with Airavata 16.0 that
> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>
> Regards,
> Shameera.
>
>
> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha  wrote:
>
>> Hello Shameera,
>> I am working on jet-stream project, where I have to find out a way to
>> submit a job in mesos/marathon cluster through Airavata client. I don't
>> have much idea from where to start looking into. Can you please give some
>> clue so that I can start working and making changes to java code for the
>> same.
>>
>> Thanks
>> Pankaj
>>
>> --
> Shameera Rathnayaka
>
>
>

Re: launching a job through Airavata to Mesos cluster

[Pierce, Marlon]

I’ll add: if submitting a job to (for example) a SLURM queuing system, we need 
to create the correct SLURM submission script and submit it by executing the 
correct command line operation (sbatch).

From: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 12:08 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>, 
Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata 
which will be run in the Mesos cluster.  I was looking for the code which 
submits jobs and wanted to make changes such a way that it can submit docker 
containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and job 
submission protocols that Shameera has mentioned. I may want to submit jobs by 
submitting a JSON through command line or any other way that you guys feel is 
more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru 
<sma...@apache.org<mailto:sma...@apache.org>> wrote:
Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled 
by Mesos? Or do you just need a client which will connect to Airavata hosted 
and managed by Mesos/Marathon?

Suresh

On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka 
<shameerai...@gmail.com<mailto:shameerai...@gmail.com>> wrote:

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer 
protocol? What is the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new 
Provider implementation to submit the request to Mesos/Marathon cluster.  But 
if you are using Airavata 16.0 which is current master, then you need to write 
JobSubmissionTask implementation. Either case you can go through the existing 
implementations, for Provider implementation see  SSHProvider  and 
JobSubmissionTask implementation see SSHJobSubmissionTask.

If I get the answers to my questions then i can provide exactly what you need 
to do. BTW we have cleaned our internal architecture in Airavata 16.0, as a 
developer you would find it easy to work with Airavata 16.0 that Airavata 15.0. 
But notice master is not yet stable as Airavata 15.0.

Regards,
Shameera.


On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>> wrote:
Hello Shameera,
I am working on jet-stream project, where I have to find out a way to submit a 
job in mesos/marathon cluster through Airavata client. I don't have much idea 
from where to start looking into. Can you please give some clue so that I can 
start working and making changes to java code for the same.

Thanks
Pankaj

--
Shameera Rathnayaka

Re: launching a job through Airavata to Mesos cluster

[Pierce, Marlon]

Hi Pankaj—

I may not have the full background on what you want to do, but we more 
typically use Airavata to run scientific applications on clusters. This goes 
through a scheduling and queuing system (Torque/MOAB or SLURM).  If you want to 
do this, you would first need to say how you would submit a dockerized 
application using Mesos as your queueing system. You could do this for a Tomcat 
server, but that is a little different from the jobs we normally run.

We don’t use Airavata to bootstrap itself, although that may be an interesting 
way to think of things.  There it is interesting to use Mesos + Marathon to run 
Airavata services (which may themselves by dockerized) but that would be done 
today outside Airavata.

Marlon


From: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Date: Wednesday, October 28, 2015 at 12:25 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Cc: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj,

You cannot do this out of the box now, but with some guidance you can easily 
write a task to run  Dockerized application on a Mesos cluster, please use 0.16 
for this. As Shameera explained this is a improvised architecture and any 
contributions you will make will carry forward. Note: This will require you to 
understand under the hood details of Airavata Architecture but I am sure you 
will motivate others to work with you to speed it up.

Suresh

On Oct 28, 2015, at 12:22 PM, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>> wrote:

“launch a dockerized job”  means submitting a docker container for any specific 
job (say a java webserver docker container) through Airavata.
What I remember Airavata php gateway can not submit any docker 
container(correct me if I am wrong). What I want to figure out initially is 
whether its feasible to submit a docker container with help of Airavata into a 
Mesos cluster.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon 
<marpi...@iu.edu<mailto:marpi...@iu.edu>> wrote:
I’ll add: if submitting a job to (for example) a SLURM queuing system, we need 
to create the correct SLURM submission script and submit it by executing the 
correct command line operation (sbatch).

From: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 12:08 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>, 
Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>>

Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata 
which will be run in the Mesos cluster.  I was looking for the code which 
submits jobs and wanted to make changes such a way that it can submit docker 
containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and job 
submission protocols that Shameera has mentioned. I may want to submit jobs by 
submitting a JSON through command line or any other way that you guys feel is 
more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru 
<sma...@apache.org<mailto:sma...@apache.org>> wrote:
Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled 
by Mesos? Or do you just need a client which will connect to Airavata hosted 
and managed by Mesos/Marathon?

Suresh

On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka 
<shameerai...@gmail.com<mailto:shameerai...@gmail.com>> wrote:

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer 
protocol? What is the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new 
Provider implementation to submit the request to Mesos/Marathon cluster.  But 
if you are using Airavata 16.0 which is current master, then you need to write 
JobSubmissionTask implementation. Either case you can go through the existing 
implemen

Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Hi Marlon,
"we more typically use Airavata to run scientific applications on clusters"
- Our plan is to run dockerized scientific application through Airavata.
Definitely our target is not to run a Tomcat webserver, I just said that as
an example.

Give me some time, after I discuss with Prof. Madhu and Renan I will try to
reply your queries in details.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu> wrote:

> Hi Pankaj—
>
> I may not have the full background on what you want to do, but we more
> typically use Airavata to run scientific applications on clusters. This
> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
> you want to do this, you would first need to say how you would submit a
> dockerized application using Mesos as your queueing system. You could do
> this for a Tomcat server, but that is a little different from the jobs we
> normally run.
>
> We don’t use Airavata to bootstrap itself, although that may be an
> interesting way to think of things.  There it is interesting to use Mesos +
> Marathon to run Airavata services (which may themselves by dockerized) but
> that would be done today outside Airavata.
>
> Marlon
>
>
> From: Suresh Marru <sma...@apache.org>
> Date: Wednesday, October 28, 2015 at 12:25 PM
> To: Airavata Dev <dev@airavata.apache.org>
> Cc: Marlon Pierce <marpi...@iu.edu>
>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> Hi Pankaj,
>
> You cannot do this out of the box now, but with some guidance you can
> easily write a task to run  Dockerized application on a Mesos cluster,
> please use 0.16 for this. As Shameera explained this is a improvised
> architecture and any contributions you will make will carry forward. Note:
> This will require you to understand under the hood details of Airavata
> Architecture but I am sure you will motivate others to work with you to
> speed it up.
>
> Suresh
>
> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote:
>
> “launch a dockerized job”  means submitting a docker container for any
> specific job (say a java webserver docker container) through Airavata.
> What I remember Airavata php gateway can not submit any docker
> container(correct me if I am wrong). What I want to figure out initially is
> whether its feasible to submit a docker container with help of Airavata
> into a Mesos cluster.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>
>> I’ll add: if submitting a job to (for example) a SLURM queuing system, we
>> need to create the correct SLURM submission script and submit it by
>> executing the correct command line operation (sbatch).
>>
>> From: Marlon Pierce <marpi...@iu.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 12:08 PM
>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>> sma...@apache.org>, Pankaj Saha <psa...@binghamton.edu>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>> job”?
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <psa...@binghamton.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 11:56 AM
>> To: Suresh Marru <sma...@apache.org>
>> Cc: Airavata Dev <dev@airavata.apache.org>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Suresh,
>>
>> My initial understanding is, I have to launch a dockerized job through
>> Airavata which will be run in the Mesos cluster.  I was looking for the
>> code which submits jobs and wanted to make changes such a way that it can
>> submit docker containers to Mesos/Marathon cluster.
>>
>> I can use 0.15 branch and I have no idea about data transfer protocol and
>> job submission protocols that Shameera has mentioned. I may want to submit
>> jobs by submitting a JSON through command line or any other way that you
>> guys feel is more appropriate.
>>
>> I can talk to Prof. Madhu and let you know more on the requirement.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sma...@apache.org> wrote:
>>
>>> Pankaj can you clarify the following:
>>>
>>> Do you want an Airavata instance to run some dockerized applications
>>> scheduled by Mesos? Or do you just need a client which will connect 

Re: launching a job through Airavata to Mesos cluster

[Pierce, Marlon]

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata 
which will be run in the Mesos cluster.  I was looking for the code which 
submits jobs and wanted to make changes such a way that it can submit docker 
containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and job 
submission protocols that Shameera has mentioned. I may want to submit jobs by 
submitting a JSON through command line or any other way that you guys feel is 
more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru 
<sma...@apache.org<mailto:sma...@apache.org>> wrote:
Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled 
by Mesos? Or do you just need a client which will connect to Airavata hosted 
and managed by Mesos/Marathon?

Suresh

On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka 
<shameerai...@gmail.com<mailto:shameerai...@gmail.com>> wrote:

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer 
protocol? What is the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new 
Provider implementation to submit the request to Mesos/Marathon cluster.  But 
if you are using Airavata 16.0 which is current master, then you need to write 
JobSubmissionTask implementation. Either case you can go through the existing 
implementations, for Provider implementation see  SSHProvider  and 
JobSubmissionTask implementation see SSHJobSubmissionTask.

If I get the answers to my questions then i can provide exactly what you need 
to do. BTW we have cleaned our internal architecture in Airavata 16.0, as a 
developer you would find it easy to work with Airavata 16.0 that Airavata 15.0. 
But notice master is not yet stable as Airavata 15.0.

Regards,
Shameera.


On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>> wrote:
Hello Shameera,
I am working on jet-stream project, where I have to find out a way to submit a 
job in mesos/marathon cluster through Airavata client. I don't have much idea 
from where to start looking into. Can you please give some clue so that I can 
start working and making changes to java code for the same.

Thanks
Pankaj

--
Shameera Rathnayaka

Re: launching a job through Airavata to Mesos cluster

[Pamidighantam, Sudhakar V]

Pankaj:

If mesos is a scheduler, perhaps Airavata could be enhanced to submit jobs to 
using mesos scheduler ( similar to slurm/pbs/lsf scheduler). Is this what you 
are referring to?

Thanks,
Sudhakar.
On Oct 28, 2015, at 11:11 AM, Pierce, Marlon 
<marpi...@iu.edu<mailto:marpi...@iu.edu>> wrote:

I’ll add: if submitting a job to (for example) a SLURM queuing system, we need 
to create the correct SLURM submission script and submit it by executing the 
correct command line operation (sbatch).

From: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 12:08 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>, 
Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata 
which will be run in the Mesos cluster.  I was looking for the code which 
submits jobs and wanted to make changes such a way that it can submit docker 
containers to Mesos/Marathon cluster.

I can use 0.15 branch and I have no idea about data transfer protocol and job 
submission protocols that Shameera has mentioned. I may want to submit jobs by 
submitting a JSON through command line or any other way that you guys feel is 
more appropriate.

I can talk to Prof. Madhu and let you know more on the requirement.

Thanks
Pankaj







On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru 
<sma...@apache.org<mailto:sma...@apache.org>> wrote:
Pankaj can you clarify the following:

Do you want an Airavata instance to run some dockerized applications scheduled 
by Mesos? Or do you just need a client which will connect to Airavata hosted 
and managed by Mesos/Marathon?

Suresh

On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka 
<shameerai...@gmail.com<mailto:shameerai...@gmail.com>> wrote:

Hi Pankaj,

Wich version of Airavata you are working on?  what is the data transfer 
protocol? What is the job submission protocol?

Short answer:  if you are using Airavata 0.15 then you need to write new 
Provider implementation to submit the request to Mesos/Marathon cluster.  But 
if you are using Airavata 16.0 which is current master, then you need to write 
JobSubmissionTask implementation. Either case you can go through the existing 
implementations, for Provider implementation see  SSHProvider  and 
JobSubmissionTask implementation see SSHJobSubmissionTask.

If I get the answers to my questions then i can provide exactly what you need 
to do. BTW we have cleaned our internal architecture in Airavata 16.0, as a 
developer you would find it easy to work with Airavata 16.0 that Airavata 15.0. 
But notice master is not yet stable as Airavata 15.0.

Regards,
Shameera.


On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>> wrote:
Hello Shameera,
I am working on jet-stream project, where I have to find out a way to submit a 
job in mesos/marathon cluster through Airavata client. I don't have much idea 
from where to start looking into. Can you please give some clue so that I can 
start working and making changes to java code for the same.

Thanks
Pankaj

--
Shameera Rathnayaka

Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Yes Suresh that's the plan.

On Wed, Oct 28, 2015 at 12:25 PM, Suresh Marru <sma...@apache.org> wrote:

> Hi Pankaj,
>
> You cannot do this out of the box now, but with some guidance you can
> easily write a task to run  Dockerized application on a Mesos cluster,
> please use 0.16 for this. As Shameera explained this is a improvised
> architecture and any contributions you will make will carry forward. Note:
> This will require you to understand under the hood details of Airavata
> Architecture but I am sure you will motivate others to work with you to
> speed it up.
>
> Suresh
>
> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote:
>
> “launch a dockerized job”  means submitting a docker container for any
> specific job (say a java webserver docker container) through Airavata.
> What I remember Airavata php gateway can not submit any docker
> container(correct me if I am wrong). What I want to figure out initially is
> whether its feasible to submit a docker container with help of Airavata
> into a Mesos cluster.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>
>> I’ll add: if submitting a job to (for example) a SLURM queuing system, we
>> need to create the correct SLURM submission script and submit it by
>> executing the correct command line operation (sbatch).
>>
>> From: Marlon Pierce <marpi...@iu.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 12:08 PM
>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>> sma...@apache.org>, Pankaj Saha <psa...@binghamton.edu>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>> job”?
>>
>> Marlon
>>
>>
>> From: Pankaj Saha <psa...@binghamton.edu>
>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>> Date: Wednesday, October 28, 2015 at 11:56 AM
>> To: Suresh Marru <sma...@apache.org>
>> Cc: Airavata Dev <dev@airavata.apache.org>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Suresh,
>>
>> My initial understanding is, I have to launch a dockerized job through
>> Airavata which will be run in the Mesos cluster.  I was looking for the
>> code which submits jobs and wanted to make changes such a way that it can
>> submit docker containers to Mesos/Marathon cluster.
>>
>> I can use 0.15 branch and I have no idea about data transfer protocol and
>> job submission protocols that Shameera has mentioned. I may want to submit
>> jobs by submitting a JSON through command line or any other way that you
>> guys feel is more appropriate.
>>
>> I can talk to Prof. Madhu and let you know more on the requirement.
>>
>> Thanks
>> Pankaj
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Oct 28, 2015 at 11:23 AM, Suresh Marru <sma...@apache.org> wrote:
>>
>>> Pankaj can you clarify the following:
>>>
>>> Do you want an Airavata instance to run some dockerized applications
>>> scheduled by Mesos? Or do you just need a client which will connect to
>>> Airavata hosted and managed by Mesos/Marathon?
>>>
>>> Suresh
>>>
>>> On Oct 28, 2015, at 10:50 AM, Shameera Rathnayaka <
>>> shameerai...@gmail.com> wrote:
>>>
>>> Hi Pankaj,
>>>
>>> Wich version of Airavata you are working on?  what is the data transfer
>>> protocol? What is the job submission protocol?
>>>
>>> Short answer:  if you are using Airavata 0.15 then you need to write new
>>> Provider implementation to submit the request to Mesos/Marathon cluster.
>>> But if you are using Airavata 16.0 which is current master, then you need
>>> to write JobSubmissionTask implementation. Either case you can go through
>>> the existing implementations, for Provider implementation see  SSHProvider
>>>  and JobSubmissionTask implementation see SSHJobSubmissionTask.
>>>
>>> If I get the answers to my questions then i can provide exactly what you
>>> need to do. BTW we have cleaned our internal architecture in Airavata 16.0,
>>> as a developer you would find it easy to work with Airavata 16.0 that
>>> Airavata 15.0. But notice master is not yet stable as Airavata 15.0.
>>>
>>> Regards,
>>> Shameera.
>>>
>>>
>>> On Tue, Oct 27, 2015 at 1:53 PM Pankaj Saha <psa...@binghamton.edu>
>>> wrote:
>>>
>>>> Hello Shameera,
>>>> I am working on jet-stream project, where I have to find out a way to
>>>> submit a job in mesos/marathon cluster through Airavata client. I don't
>>>> have much idea from where to start looking into. Can you please give some
>>>> clue so that I can start working and making changes to java code for the
>>>> same.
>>>>
>>>> Thanks
>>>> Pankaj
>>>>
>>>> --
>>> Shameera Rathnayaka
>>>
>>>
>>>
>>
>
>

Re: launching a job through Airavata to Mesos cluster

[Pierce, Marlon]

The “hello world” for us is usually to just run some linux command on the 
remote queuing system. So that is something you could easily wrap via docker 
and submit via Mesos (I hope it is easy…).  Related to this is determining that 
the job has gone through various states (queued, executing, completed) and 
communicating that back to Airavata.


Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Date: Wednesday, October 28, 2015 at 12:54 PM
To: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Cc: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>, Airavata Dev 
<dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Marlon,
"we more typically use Airavata to run scientific applications on clusters" - 
Our plan is to run dockerized scientific application through Airavata.
Definitely our target is not to run a Tomcat webserver, I just said that as an 
example.

Give me some time, after I discuss with Prof. Madhu and Renan I will try to 
reply your queries in details.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon 
<marpi...@iu.edu<mailto:marpi...@iu.edu>> wrote:
Hi Pankaj—

I may not have the full background on what you want to do, but we more 
typically use Airavata to run scientific applications on clusters. This goes 
through a scheduling and queuing system (Torque/MOAB or SLURM).  If you want to 
do this, you would first need to say how you would submit a dockerized 
application using Mesos as your queueing system. You could do this for a Tomcat 
server, but that is a little different from the jobs we normally run.

We don’t use Airavata to bootstrap itself, although that may be an interesting 
way to think of things.  There it is interesting to use Mesos + Marathon to run 
Airavata services (which may themselves by dockerized) but that would be done 
today outside Airavata.

Marlon


From: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Date: Wednesday, October 28, 2015 at 12:25 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Cc: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>

Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj,

You cannot do this out of the box now, but with some guidance you can easily 
write a task to run  Dockerized application on a Mesos cluster, please use 0.16 
for this. As Shameera explained this is a improvised architecture and any 
contributions you will make will carry forward. Note: This will require you to 
understand under the hood details of Airavata Architecture but I am sure you 
will motivate others to work with you to speed it up.

Suresh

On Oct 28, 2015, at 12:22 PM, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>> wrote:

“launch a dockerized job”  means submitting a docker container for any specific 
job (say a java webserver docker container) through Airavata.
What I remember Airavata php gateway can not submit any docker 
container(correct me if I am wrong). What I want to figure out initially is 
whether its feasible to submit a docker container with help of Airavata into a 
Mesos cluster.

Thanks
Pankaj

On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon 
<marpi...@iu.edu<mailto:marpi...@iu.edu>> wrote:
I’ll add: if submitting a job to (for example) a SLURM queuing system, we need 
to create the correct SLURM submission script and submit it by executing the 
correct command line operation (sbatch).

From: Marlon Pierce <marpi...@iu.edu<mailto:marpi...@iu.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 12:08 PM
To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>, 
Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>, Pankaj Saha 
<psa...@binghamton.edu<mailto:psa...@binghamton.edu>>

Subject: Re: launching a job through Airavata to Mesos cluster

Hi Pankaj, can you say more about what you mean by “launch a dockerized job”?

Marlon


From: Pankaj Saha <psa...@binghamton.edu<mailto:psa...@binghamton.edu>>
Reply-To: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Wednesday, October 28, 2015 at 11:56 AM
To: Suresh Marru <sma...@apache.org<mailto:sma...@apache.org>>
Cc: Airavata Dev <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Re: launching a job through Airavata to Mesos cluster

Hi Suresh,

My initial understanding is, I have to launch a dockerized job through Airavata 
which will be run in the Mesos cluster.  I was looking for the code which 
submits jobs and wanted to make changes such a way that it can submit docker 
containers to Mesos/Marathon clust

Re: launching a job through Airavata to Mesos cluster

[Pankaj Saha]

As Suresh said I think this is the time  when I have to understand under
the hood details of Airavata Architecture, which I could not gain much
during GSoC.

On Wed, Oct 28, 2015 at 1:00 PM, Pierce, Marlon <marpi...@iu.edu> wrote:

> The “hello world” for us is usually to just run some linux command on the
> remote queuing system. So that is something you could easily wrap via
> docker and submit via Mesos (I hope it is easy…).  Related to this is
> determining that the job has gone through various states (queued,
> executing, completed) and communicating that back to Airavata.
>
>
> Marlon
>
>
> From: Pankaj Saha <psa...@binghamton.edu>
> Date: Wednesday, October 28, 2015 at 12:54 PM
> To: Marlon Pierce <marpi...@iu.edu>
> Cc: Suresh Marru <sma...@apache.org>, Airavata Dev <
> dev@airavata.apache.org>
>
> Subject: Re: launching a job through Airavata to Mesos cluster
>
> Hi Marlon,
> "we more typically use Airavata to run scientific applications on
> clusters" - Our plan is to run dockerized scientific application through A
> iravata.
> Definitely our target is not to run a Tomcat webserver, I just said that
> as an example.
>
> Give me some time, after I discuss with Prof. Madhu and Renan I will try
> to reply your queries in details.
>
> Thanks
> Pankaj
>
> On Wed, Oct 28, 2015 at 12:32 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>
>> Hi Pankaj—
>>
>> I may not have the full background on what you want to do, but we more
>> typically use Airavata to run scientific applications on clusters. This
>> goes through a scheduling and queuing system (Torque/MOAB or SLURM).  If
>> you want to do this, you would first need to say how you would submit a
>> dockerized application using Mesos as your queueing system. You could do
>> this for a Tomcat server, but that is a little different from the jobs we
>> normally run.
>>
>> We don’t use Airavata to bootstrap itself, although that may be an
>> interesting way to think of things.  There it is interesting to use Mesos +
>> Marathon to run Airavata services (which may themselves by dockerized) but
>> that would be done today outside Airavata.
>>
>> Marlon
>>
>>
>> From: Suresh Marru <sma...@apache.org>
>> Date: Wednesday, October 28, 2015 at 12:25 PM
>> To: Airavata Dev <dev@airavata.apache.org>
>> Cc: Marlon Pierce <marpi...@iu.edu>
>>
>> Subject: Re: launching a job through Airavata to Mesos cluster
>>
>> Hi Pankaj,
>>
>> You cannot do this out of the box now, but with some guidance you can
>> easily write a task to run  Dockerized application on a Mesos cluster,
>> please use 0.16 for this. As Shameera explained this is a improvised
>> architecture and any contributions you will make will carry forward. Note:
>> This will require you to understand under the hood details of Airavata
>> Architecture but I am sure you will motivate others to work with you to
>> speed it up.
>>
>> Suresh
>>
>> On Oct 28, 2015, at 12:22 PM, Pankaj Saha <psa...@binghamton.edu> wrote:
>>
>> “launch a dockerized job”  means submitting a docker container for any
>> specific job (say a java webserver docker container) through Airavata.
>> What I remember Airavata php gateway can not submit any docker
>> container(correct me if I am wrong). What I want to figure out initially is
>> whether its feasible to submit a docker container with help of Airavata
>> into a Mesos cluster.
>>
>> Thanks
>> Pankaj
>>
>> On Wed, Oct 28, 2015 at 12:11 PM, Pierce, Marlon <marpi...@iu.edu> wrote:
>>
>>> I’ll add: if submitting a job to (for example) a SLURM queuing system,
>>> we need to create the correct SLURM submission script and submit it by
>>> executing the correct command line operation (sbatch).
>>>
>>> From: Marlon Pierce <marpi...@iu.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 12:08 PM
>>> To: Airavata Dev <dev@airavata.apache.org>, Suresh Marru <
>>> sma...@apache.org>, Pankaj Saha <psa...@binghamton.edu>
>>>
>>> Subject: Re: launching a job through Airavata to Mesos cluster
>>>
>>> Hi Pankaj, can you say more about what you mean by “launch a dockerized
>>> job”?
>>>
>>> Marlon
>>>
>>>
>>> From: Pankaj Saha <psa...@binghamton.edu>
>>> Reply-To: Airavata Dev <dev@airavata.apache.org>
>>> Date: Wednesday, October 28, 2015 at 11:56 AM
>>&g

launching a job through Airavata to Mesos cluster

[Pankaj Saha]

Hello Shameera,
I am working on jet-stream project, where I have to find out a way to
submit a job in mesos/marathon cluster through Airavata client. I don't
have much idea from where to start looking into. Can you please give some
clue so that I can start working and making changes to java code for the
same.

Thanks
Pankaj