Thanks Mangirish for your contribution, this is very neat implementation. Renan, Pankaj,
Can we get on a google hangout and brainstorm on how to integrate the Auroa/Mesos work you are doing with the Open Stack integration Mangirish has contributed to. Suresh > On Apr 16, 2016, at 12:57 AM, Mangirish Wagle <[email protected]> > wrote: > > Hello Team, > > I have created a new pull request with the changes that I added today:- > https://github.com/apache/airavata/pull/32 > <https://github.com/apache/airavata/pull/32> > > Following are the main changes added with this request:- > 1) Added method to Cloud Interface to associate floating ip. The floating ip > will also get deallocated on deletion of server instance. > 2) Changed the methods to use network name instead of network id, read from > the properties, for better understanding. > 3) Added log statements in the Interface implementation for OpenStack > (Jetstream). > > Thanks. > > Regards, > Mangirish > > On Tue, Apr 12, 2016 at 12:56 AM, Mangirish Wagle <[email protected] > <mailto:[email protected]>> wrote: > Hello, > > I have created a new pull request for cloud-provisioning project after making > all the changes suggested during the code review conducted today during the > meeting with Suresh and Shameera. Following is the link:- > https://github.com/apache/airavata/pull/31 > <https://github.com/apache/airavata/pull/31> > > Also, for the team's awareness, we have managed to configure a new network > topology in the Jetstream Openstack cloud. The name of the network is > "airavata" and it is connected to the "public" network using a router. This > now enables us to provision instances and associate publicly accessible > floating IPs so that they are accessible (over ssh) from Internet. > > Thanks. > > Best Regards, > Mangirish > > On Wed, Apr 6, 2016 at 12:08 AM, Mangirish Wagle <[email protected] > <mailto:[email protected]>> wrote: > Hello, > > I have managed to put together a Cloud Interface project as initial POC with > utility functions to create, delete servers. I have created a common cloud > interface which has been implemented for Openstack Clouds using Openstack4j. > > A maven build has been setup for the project and a sample unit test has been > added to the project to test and demonstrate a server create with associated > keypair and delete operation on Jetstream Openstack using scigap credentials. > A README file added to the project contain the steps to test run the project. > > The current code does not handle the network setup that is required to make > the virtual machines created, accessible over the public network. I shall > work on getting this done as soon as I find some time out of my academic > activities and schedule. > > I have created following pull request for the current code from my forked > repo to Airavata repo:- > > https://github.com/apache/airavata/pull/30 > <https://github.com/apache/airavata/pull/30> > > You may please review and let me know your comments. > > Thanks. > > Best Regards, > Mangirish > > > On Thu, Mar 24, 2016 at 9:42 PM, Suresh Marru <[email protected] > <mailto:[email protected]>> wrote: > Hi Mangirish, > > Yes now I noticed the scaling within the heat section. Yes it makes sense to > leave it behind the orchestration layer not to re-invent that logic. > > Airavata Orchestrator will be the natural plan to call the provisioning > service and bootstrap the mesos cluster. The ansible I referred to are not > yet contributed into the repo. I am cc’ing Pankaj and Renan who can probably > make that contribution. You can read about their effort in > http://onlinelibrary.wiley.com/doi/10.1002/cpe.3708/full > <http://onlinelibrary.wiley.com/doi/10.1002/cpe.3708/full> > > Renan, > > Mangirish is proposing a project to programmatically interact with Cloud > Interfaces (like Open Stack on Jetstream) and provision resources. I would > assume then the component you have developed will take over and bootstrap the > mesos cluster which GFac can then submit jobs to (through Aurora). > > Suresh > > >> On Mar 24, 2016, at 9:14 PM, Mangirish Wagle <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hello, >> >> I was trying to understand the end result flow of the Airavata with Cloud >> Orchestrator and had the following question:- >> >> Once the cluster has been setup, as we discussed, an ansible or some >> configuration management tool would boostrap and configure mesos. Which >> component in Airavata would host and call the ansible script and what event >> would trigger it? >> >> Thanks. >> >> Regards, >> Mangirish >> >> On Thu, Mar 24, 2016 at 9:07 PM, Mangirish Wagle <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks for your feedback Suresh! >> >> I have mentioned about the Autoscaling in the Heat Orchestration solution, >> which does the dynamic scaling of resources in an existing cloud. Please let >> me know if you think that needs to be restructured. >> >> Also, I have updated the Google doc and Wiki with the revised proposal, >> after making changes as per Marlon's review comments. >> >> I request you to please review again and check if there is anything that >> needs still needs to be revised. >> >> Thank you! >> >> Regards, >> Mangirish >> >> On Thu, Mar 24, 2016 at 7:18 PM, Suresh Marru <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Mangirish, >> >> Your proposal has all the required good detail. One optional addition you >> can clarify on if you can expand or contract resources to a previously >> provisioned cloud. >> >> Suresh >> >>> On Mar 23, 2016, at 9:10 PM, Mangirish Wagle <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Thanks Shameera for the info and sharing the JIRA Epic details. >>> >>> I have drafted my GSOC Proposal for the project and I request you to please >>> review the same:- >>> >>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata >>> >>> <https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+Proposal-+Cloud+Based+Clusters+for+Apache+Airavata> >>> >>> I shall submit this on the GSOC portal by tomorrow, once I get my >>> enrollment verification proof. >>> >>> Regards, >>> Mangirish >>> >>> >>> >>> On Wed, Mar 23, 2016 at 12:29 PM, Shameera Rathnayaka >>> <[email protected] <mailto:[email protected]>> wrote: >>> Hi Mangirish, >>> >>> Yes your above understanding is right. Gfac is like task executor which >>> execute what ever task given by Orchestrator. >>> >>> Here is the epic https://issues.apache.org/jira/browse/AIRAVATA-1924 >>> <https://issues.apache.org/jira/browse/AIRAVATA-1924>, Open stack >>> integration is part of this epic, you can create a new top level jira >>> ticket and create subtask under that ticket. >>> >>> Regards, >>> Shameera. >>> >>> On Wed, Mar 23, 2016 at 12:20 PM Mangirish Wagle <[email protected] >>> <mailto:[email protected]>> wrote: >>> Thanks Marlon for the info. So what I get is that the Orchestrator would >>> decide if the job needs to be submitted to cloud based cluster and route it >>> to GFAC which would have a separate interfacing with the cloud cluster >>> service. >>> >>> Also I wanted to know if there is any Story/ Epic created in JIRA for this >>> project which I can use to create and track tasks? If not can I create one? >>> >>> Thanks. >>> >>> Regards, >>> Mangirish >>> >>> On Wed, Mar 23, 2016 at 12:01 PM, Pierce, Marlon <[email protected] >>> <mailto:[email protected]>> wrote: >>> The Application Factory component is called “gfac” in the code base. This >>> is the part that handles the interfacing to the remote resource (most often >>> by ssh but other providers exist). The Orchestrator routes jobs to GFAC >>> instances. >>> >>> From: Mangirish Wagle <[email protected] >>> <mailto:[email protected]>> >>> Reply-To: "[email protected] <mailto:[email protected]>" >>> <[email protected] <mailto:[email protected]>> >>> Date: Wednesday, March 23, 2016 at 11:56 AM >>> To: "[email protected] <mailto:[email protected]>" >>> <[email protected] <mailto:[email protected]>> >>> Subject: Re: [GSOC Proposal] Cloud based clusters for Apache Airavata >>> >>> Hello Team, >>> >>> I was drafting the GSOC proposal and I just had a quick question about the >>> integration of the project with Apache Airavata. >>> >>> Which is the component in Airavata that would call the service to provision >>> the cloud cluster? >>> >>> I am looking at the Airavata architecture diagram and my understanding is >>> that this would be treated as a new Application and would have a separate >>> application interface in 'Application Factory' component. Also the workflow >>> orchestrator would be having the intelligence to figure out which jobs to >>> be submitted to cloud based clusters. >>> >>> Please let me know whether my understanding is correct. >>> >>> Thank you. >>> >>> Best Regards, >>> Mangirish Wagle >>> >>> On Tue, Mar 22, 2016 at 2:28 PM, Pierce, Marlon <[email protected] >>> <mailto:[email protected]>> wrote: >>> Hi Mangirish, please add your proposal to the GSOC 2016 site. >>> >>> From: Mangirish Wagle <[email protected] >>> <mailto:[email protected]>> >>> Reply-To: "[email protected] <mailto:[email protected]>" >>> <[email protected] <mailto:[email protected]>> >>> Date: Thursday, March 17, 2016 at 3:35 PM >>> To: "[email protected] <mailto:[email protected]>" >>> <[email protected] <mailto:[email protected]>> >>> Subject: [GSOC Proposal] Cloud based clusters for Apache Airavata >>> >>> Hello Dev Team, >>> >>> I had the opportunity to interact with Suresh and Shameera wherein we >>> discussed an open requirement in Airavata to be addressed. The requirement >>> is to expand the capabilities of Apache Airavata to submit jobs to cloud >>> based clusters in addition to HPC/ HTC clusters. >>> >>> The idea is to dynamically provision a cloud cluster in an environment like >>> Jetstream, based on the configuration figured out by Airavata, which would >>> be operated by a distributed system management software like Mesos. An >>> initial high level goals would be:- >>> Airavata categorizes certain jobs to be run on cloud based clusters and >>> figure out the required hardware config for the cluster. >>> The proposed service would provision the cluster with the required >>> resources. >>> An ansible script would configure a Mesos cluster with the resources >>> provisioned. >>> Airavata submits the job to the Mesos cluster. >>> Mesos then figures out the efficient resource allocation within the cluster >>> and runs the job and fetches the result. >>> The cluster is then deprovisioned automatically when not in use. >>> The project would mainly focus on point 2 and 6 above. >>> >>> To start with, I am currently trying to get a working prototype of setting >>> up compute nodes on an openstack environment using JClouds (Targetted for >>> Jetstream). Also, I am planning to explore the option of using Openstack >>> Heat engine to orchestrate the cluster. However, going ahead Airavata would >>> be supporting other clouds like Amazon EC2 or Comet cluster, so we need to >>> have a generic solution for achieving the goal. >>> >>> Another approach which might be efficient in terms of performance and time >>> is using a container based clouds using Docker, Kubernetes which would have >>> substantially less bootstrap time compared to cloud VMs. This would be a >>> future prospect as we may not have all the clusters supporting >>> containerization. >>> >>> This has been considered as a potential GSOC project and I would be working >>> on drafting a proposal on this idea. >>> >>> Any inputs/ comments/ suggestions would be very helpful. >>> >>> Best Regards, >>> Mangirish Wagle >>> >>> >>> -- >>> Shameera Rathnayaka >>> >> >> >> > > > >
