Re: Re: Note to potential GSOC students

[Mangirish Wagle]

Hi Marlon,

I have submitted the first draft of my proposal as per your guidelines to
ASF with Apache Airavata in the title, in the GSOC portal.
Please let me know if you can see my draft.

Thank you.

Regards,
Mangirish Wagle

On Wed, Mar 23, 2016 at 11:49 AM, Jatin Balodhi 
wrote:

> There's an option of " Apache Software Foundation proposal tag" what
> should i select there?
>
> Thanks
> Jatin
>
>  Forwarded Message 
> Subject: Re: Note to potential GSOC students
> Date: Wed, 23 Mar 2016 15:36:58 +
> From: Pierce, Marlon  
> To: Jatin Balodhi  
>
> Hi Jatin, you have not yet started your application to GSOC program, as
> far as I can tell.  I do not see your application in the project listings.
> Do this first. Make sure Apache Airavata is in your title.
>
> Marlon
>
>
> From: Jatin Balodhi 
> Reply-To: " dev@airavata.apache.org" <
> dev@airavata.apache.org>
> Date: Wednesday, March 23, 2016 at 11:17 AM
> To: " dev@airavata.apache.org" <
> dev@airavata.apache.org>
> Subject: Re: Note to potential GSOC students
>
> Hi Marlon,
>
> I made some changes to my GSOC proposal as you said, can you look at it
> once more?
>
> Thanks
> Jatin
>
> On Wednesday 23 March 2016 07:52 PM, Pierce, Marlon wrote:
>
> Please make sure you have started the proposal submission process
> correctly in the GSOC site,  
> https://summerofcode.withgoogle.com/.  I see proposal drafts for only
> about half of those students who have expressed interest.
>
> Thanks,
>
> Marlon
>
>
>
>
>

RE: [GSoC Proposal] In-Situ Simulation Monitoring and Analysis Using Apache Airavata

[Miller, Mark]

I like this proposal also very much, and agree with what Sandy said as well.

Mark


From: Sandra Gesing [mailto:sandra.ges...@nd.edu]
Sent: Tuesday, March 22, 2016 6:12 PM
To: dev@airavata.apache.org
Subject: Re: [GSoC Proposal] In-Situ Simulation Monitoring and Analysis Using 
Apache Airavata

Hi Jeff,

the proposal looks already excellent. I think it would be beneficial to have 
the feature in Airavata that users can directly search for certain terms in a 
log file and intermediate results additionally to see the whole log file and 
intermediate values.
Thanks,
Sandra


Sandra Gesing
Research Assistant Professor, Department of Computer Science and Engineering
Computational Scientist, Center for Research Computing
University of Notre Dame
http://www3.nd.edu/~sgesing


On Tue, Mar 22, 2016 at 8:03 PM, Jeffery Kinnison 
> wrote:
Hello Dev Team,
I have created a draft of my GSoC proposal and made it available as at the link 
below.

https://docs.google.com/document/d/16BCTV__tA9h2Pnx1annaBIJaU2Jmma1ayTHV6BNMzNA/edit?usp=sharing
I would appreciate any comments to improve this proposal.
Best,
Jeff Kinnison

Re: [GSOC Proposal] Cloud based clusters for Apache Airavata

[Shameera Rathnayaka]

Hi Mangirish,

Yes your above understanding is right. Gfac is like task executor which
execute what ever task given by Orchestrator.

Here is the epic https://issues.apache.org/jira/browse/AIRAVATA-1924, Open
stack integration is part of this epic, you can create a new top level jira
ticket and create subtask under that ticket.

Regards,
Shameera.

On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle 
wrote:

> Thanks Marlon for the info. So what I get is that the Orchestrator would
> decide if the job needs to be submitted to cloud based cluster and route it
> to GFAC which would have a separate interfacing with the cloud cluster
> service.
>
> Also I wanted to know if there is any Story/ Epic created in JIRA for this
> project which I can use to create and track tasks? If not can I create one?
>
> Thanks.
>
> Regards,
> Mangirish
>
> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon  wrote:
>
>> The Application Factory component is called “gfac” in the code base.
>> This is the part that handles the interfacing to the remote resource (most
>> often by ssh but other providers exist). The Orchestrator routes jobs to
>> GFAC instances.
>>
>> From: Mangirish Wagle 
>> Reply-To: "dev@airavata.apache.org" 
>> Date: Wednesday, March 23, 2016 at 11:56 AM
>> To: "dev@airavata.apache.org" 
>> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>
>> Hello Team,
>>
>> I was drafting the GSOC proposal and I just had a quick question about
>> the integration of the project with Apache Airavata.
>>
>> Which is the component in Airavata that would call the service to
>> provision the cloud cluster?
>>
>> I am looking at the Airavata architecture diagram and my understanding is
>> that this would be treated as a new Application and would have a separate
>> application interface in 'Application Factory' component. Also the workflow
>> orchestrator would be having the intelligence to figure out which jobs to
>> be submitted to cloud based clusters.
>>
>> Please let me know whether my understanding is correct.
>>
>> Thank you.
>>
>> Best Regards,
>> Mangirish Wagle
>>
>> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon  wrote:
>>
>>> Hi Mangirish, please add your proposal to the GSOC 2016 site.
>>>
>>> From: Mangirish Wagle 
>>> Reply-To: "dev@airavata.apache.org" 
>>> Date: Thursday, March 17, 2016 at 3:35 PM
>>> To: "dev@airavata.apache.org" 
>>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>>
>>> Hello Dev Team,
>>>
>>> I had the opportunity to interact with Suresh and Shameera wherein we
>>> discussed an open requirement in Airavata to be addressed. The requirement
>>> is to expand the capabilities of Apache Airavata to submit jobs to cloud
>>> based clusters in addition to HPC/ HTC clusters.
>>>
>>> The idea is to dynamically provision a cloud cluster in an environment
>>> like Jetstream, based on the configuration figured out by Airavata, which
>>> would be operated by a distributed system management software like Mesos.
>>> An initial high level goals would be:-
>>>
>>>1. Airavata categorizes certain jobs to be run on cloud based
>>>clusters and figure out the required hardware config for the cluster.
>>>2. The proposed service would provision the cluster with the
>>>required resources.
>>>3. An ansible script would configure a Mesos cluster with the
>>>resources provisioned.
>>>4. Airavata submits the job to the Mesos cluster.
>>>5. Mesos then figures out the efficient resource allocation within
>>>the cluster and runs the job and fetches the result.
>>>6. The cluster is then deprovisioned automatically when not in use.
>>>
>>> The project would mainly focus on point 2 and 6 above.
>>>
>>> To start with, I am currently trying to get a working prototype of
>>> setting up compute nodes on an openstack environment using JClouds
>>> (Targetted for Jetstream). Also, I am planning to explore the option of
>>> using Openstack Heat engine to orchestrate the cluster. However, going
>>> ahead Airavata would be supporting other clouds like Amazon EC2 or Comet
>>> cluster, so we need to have a generic solution for achieving the goal.
>>>
>>> Another approach which might be efficient in terms of performance and
>>> time is using a container based clouds using Docker, Kubernetes which would
>>> have substantially less bootstrap time compared to cloud VMs. This would be
>>> a future prospect as we may not have all the clusters supporting
>>> containerization.
>>>
>>> This has been considered as a potential GSOC project and I would be
>>> working on drafting a proposal on this idea.
>>>
>>> Any inputs/ comments/ suggestions would be very helpful.
>>>
>>> Best Regards,
>>> Mangirish Wagle
>>>
>>
>>
> --
Shameera Rathnayaka

Re: [GSOC Proposal] Cloud based clusters for Apache Airavata

[Mangirish Wagle]

Thanks Marlon for the info. So what I get is that the Orchestrator would
decide if the job needs to be submitted to cloud based cluster and route it
to GFAC which would have a separate interfacing with the cloud cluster
service.

Also I wanted to know if there is any Story/ Epic created in JIRA for this
project which I can use to create and track tasks? If not can I create one?

Thanks.

Regards,
Mangirish

On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon  wrote:

> The Application Factory component is called “gfac” in the code base.  This
> is the part that handles the interfacing to the remote resource (most often
> by ssh but other providers exist). The Orchestrator routes jobs to GFAC
> instances.
>
> From: Mangirish Wagle 
> Reply-To: "dev@airavata.apache.org" 
> Date: Wednesday, March 23, 2016 at 11:56 AM
> To: "dev@airavata.apache.org" 
> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata
>
> Hello Team,
>
> I was drafting the GSOC proposal and I just had a quick question about the
> integration of the project with Apache Airavata.
>
> Which is the component in Airavata that would call the service to
> provision the cloud cluster?
>
> I am looking at the Airavata architecture diagram and my understanding is
> that this would be treated as a new Application and would have a separate
> application interface in 'Application Factory' component. Also the workflow
> orchestrator would be having the intelligence to figure out which jobs to
> be submitted to cloud based clusters.
>
> Please let me know whether my understanding is correct.
>
> Thank you.
>
> Best Regards,
> Mangirish Wagle
>
> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon  wrote:
>
>> Hi Mangirish, please add your proposal to the GSOC 2016 site.
>>
>> From: Mangirish Wagle 
>> Reply-To: "dev@airavata.apache.org" 
>> Date: Thursday, March 17, 2016 at 3:35 PM
>> To: "dev@airavata.apache.org" 
>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata
>>
>> Hello Dev Team,
>>
>> I had the opportunity to interact with Suresh and Shameera wherein we
>> discussed an open requirement in Airavata to be addressed. The requirement
>> is to expand the capabilities of Apache Airavata to submit jobs to cloud
>> based clusters in addition to HPC/ HTC clusters.
>>
>> The idea is to dynamically provision a cloud cluster in an environment
>> like Jetstream, based on the configuration figured out by Airavata, which
>> would be operated by a distributed system management software like Mesos.
>> An initial high level goals would be:-
>>
>>1. Airavata categorizes certain jobs to be run on cloud based
>>clusters and figure out the required hardware config for the cluster.
>>2. The proposed service would provision the cluster with the required
>>resources.
>>3. An ansible script would configure a Mesos cluster with the
>>resources provisioned.
>>4. Airavata submits the job to the Mesos cluster.
>>5. Mesos then figures out the efficient resource allocation within
>>the cluster and runs the job and fetches the result.
>>6. The cluster is then deprovisioned automatically when not in use.
>>
>> The project would mainly focus on point 2 and 6 above.
>>
>> To start with, I am currently trying to get a working prototype of
>> setting up compute nodes on an openstack environment using JClouds
>> (Targetted for Jetstream). Also, I am planning to explore the option of
>> using Openstack Heat engine to orchestrate the cluster. However, going
>> ahead Airavata would be supporting other clouds like Amazon EC2 or Comet
>> cluster, so we need to have a generic solution for achieving the goal.
>>
>> Another approach which might be efficient in terms of performance and
>> time is using a container based clouds using Docker, Kubernetes which would
>> have substantially less bootstrap time compared to cloud VMs. This would be
>> a future prospect as we may not have all the clusters supporting
>> containerization.
>>
>> This has been considered as a potential GSOC project and I would be
>> working on drafting a proposal on this idea.
>>
>> Any inputs/ comments/ suggestions would be very helpful.
>>
>> Best Regards,
>> Mangirish Wagle
>>
>
>

Re: [GSOC Proposal] Cloud based clusters for Apache Airavata

[Pierce, Marlon]

The Application Factory component is called “gfac” in the code base.  This is 
the part that handles the interfacing to the remote resource (most often by ssh 
but other providers exist). The Orchestrator routes jobs to GFAC instances.

From: Mangirish Wagle 
>
Reply-To: "dev@airavata.apache.org" 
>
Date: Wednesday, March 23, 2016 at 11:56 AM
To: "dev@airavata.apache.org" 
>
Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata

Hello Team,

I was drafting the GSOC proposal and I just had a quick question about the 
integration of the project with Apache Airavata.

Which is the component in Airavata that would call the service to provision the 
cloud cluster?

I am looking at the Airavata architecture diagram and my understanding is that 
this would be treated as a new Application and would have a separate 
application interface in 'Application Factory' component. Also the workflow 
orchestrator would be having the intelligence to figure out which jobs to be 
submitted to cloud based clusters.

Please let me know whether my understanding is correct.

Thank you.

Best Regards,
Mangirish Wagle

On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon 
> wrote:
Hi Mangirish, please add your proposal to the GSOC 2016 site.

From: Mangirish Wagle 
>
Reply-To: "dev@airavata.apache.org" 
>
Date: Thursday, March 17, 2016 at 3:35 PM
To: "dev@airavata.apache.org" 
>
Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata

Hello Dev Team,

I had the opportunity to interact with Suresh and Shameera wherein we discussed 
an open requirement in Airavata to be addressed. The requirement is to expand 
the capabilities of Apache Airavata to submit jobs to cloud based clusters in 
addition to HPC/ HTC clusters.

The idea is to dynamically provision a cloud cluster in an environment like 
Jetstream, based on the configuration figured out by Airavata, which would be 
operated by a distributed system management software like Mesos. An initial 
high level goals would be:-

  1.  Airavata categorizes certain jobs to be run on cloud based clusters and 
figure out the required hardware config for the cluster.
  2.  The proposed service would provision the cluster with the required 
resources.
  3.  An ansible script would configure a Mesos cluster with the resources 
provisioned.
  4.  Airavata submits the job to the Mesos cluster.
  5.  Mesos then figures out the efficient resource allocation within the 
cluster and runs the job and fetches the result.
  6.  The cluster is then deprovisioned automatically when not in use.

The project would mainly focus on point 2 and 6 above.

To start with, I am currently trying to get a working prototype of setting up 
compute nodes on an openstack environment using JClouds (Targetted for 
Jetstream). Also, I am planning to explore the option of using Openstack Heat 
engine to orchestrate the cluster. However, going ahead Airavata would be 
supporting other clouds like Amazon EC2 or Comet cluster, so we need to have a 
generic solution for achieving the goal.

Another approach which might be efficient in terms of performance and time is 
using a container based clouds using Docker, Kubernetes which would have 
substantially less bootstrap time compared to cloud VMs. This would be a future 
prospect as we may not have all the clusters supporting containerization.

This has been considered as a potential GSOC project and I would be working on 
drafting a proposal on this idea.

Any inputs/ comments/ suggestions would be very helpful.

Best Regards,
Mangirish Wagle

Re: [GSoC Proposal] - Integrating Resource Information from Apache Mesos with Apache Airavata’s Job Management Modules

[Pierce, Marlon]

Hi Pankaj,

I still don’t see your application in the GSOC site. Please make sure you have 
done this correctly.

Thanks,

Marlon

From: Pankaj Saha >
Date: Monday, March 21, 2016 at 5:17 PM
To: marpierc >
Cc: "dev@airavata.apache.org" 
>
Subject: Re: [GSoC Proposal] - Integrating Resource Information from Apache 
Mesos with Apache Airavata’s Job Management Modules

Hi Marlon,
Here is the link that I have created.

https://docs.google.com/document/d/1qtFvg4-usT4D_1TDNBsQDFQGZIkH99ideYQ1T3HU9nY/edit?usp=sharing
The draft is created under the GSoC proposal site under Apache foundation with 
the same title.


Thanks
Pankaj



On Mon, Mar 21, 2016 at 4:47 PM, Pierce, Marlon 
> wrote:
Hi Pankaj,

I have some comments, but it would be easier if you created a proposal draft in 
the GSOC site. The google doc option for your draft is better than pointing to 
the Airavata wiki. Please make sure you give comment and suggestion permissions.

Marlon


From: Pankaj Saha >
Reply-To: "dev@airavata.apache.org" 
>
Date: Monday, March 21, 2016 at 11:16 AM
To: dev >
Subject: [GSoC Proposal] - Integrating Resource Information from Apache Mesos 
with Apache Airavata’s Job Management Modules

Hi Dev Team,

Please review the following GSoC proposal that I plan to submit:
Title: Integrating Resource Information from Apache Mesos with Apache 
Airavata’s Job Management Modules

Abstract:
Apache Airavata provides gateway computing capability across clustered 
environments for scientific users. It abstracts away the complexities of 
submitting jobs to HPC platforms and provides users with an intuitive and 
elegant web-based interface to submit jobs. Apache Mesos is a  distributed 
kernel that manages distributed computing resources as a single computer. As 
Airavata is being extended to use Big Data and Cloud tools to launch jobs in 
cloud environments, it needs to retrieve the resource and job execution 
information from the Big Data framework back to the Apache portal accessible to 
the end user. In this project we will develop code and scripts to be integrated 
with the Airavata that will use the HTTP API of Mesos to continuously fetch the 
complete resource and scheduling information. This information can then be used 
by Airavata to dynamically monitor and improve its job submission strategy in 
cloud environments such as Jetstream.

Introduction:
Apache Mesos provides HTTP API endpoints for scheduler, executor, internal and 
admin related queries. To fetch information regarding a clustered environment 
that is managed by the Mesos master, the API can be accessed via curl requests 
over HTTP. The response to such requests will be received as well formed json 
document. We will parse the json response and present the information in the 
format desired. The retrieved information will include resource usage, resource 
available for further jobs, job status, time elapsed since the job started, 
etc.  Airavata, in turn, will use this information to determine the resource 
usage, performance of the jobs on a job submission, rapid diagnosis on the 
health of the submitted jobs.

We will use the observer pattern to continuously pull information from Cloud 
and big Data Resource Managers, such as Apache Mesos, to Airavata.

Any comment and suggestions would be very helpful.

Thanks
Pankaj