Updated Branches: refs/heads/master a36c8a962 -> 2b0946c6d
[GSOC] Mesos integration mid-term report Signed-off-by: Sebastien Goasguen <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/2b0946c6 Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/2b0946c6 Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/2b0946c6 Branch: refs/heads/master Commit: 2b0946c6db46136b957b80203cef2418f482183e Parents: a36c8a9 Author: Sebastien Goasguen <[email protected]> Authored: Tue Jul 30 13:17:46 2013 -0400 Committer: Sebastien Goasguen <[email protected]> Committed: Tue Jul 30 13:17:46 2013 -0400 ---------------------------------------------------------------------- docs/en-US/gsoc-midsummer-dharmesh.xml | 169 +++++++++++++++++++++++++++- 1 file changed, 167 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cloudstack/blob/2b0946c6/docs/en-US/gsoc-midsummer-dharmesh.xml ---------------------------------------------------------------------- diff --git a/docs/en-US/gsoc-midsummer-dharmesh.xml b/docs/en-US/gsoc-midsummer-dharmesh.xml index 69e417a..9e0fdcf 100644 --- a/docs/en-US/gsoc-midsummer-dharmesh.xml +++ b/docs/en-US/gsoc-midsummer-dharmesh.xml @@ -23,6 +23,171 @@ --> <section id="gsoc-midsummer-dharmesh"> - <title>Mid-Summer Progress Updates</title> - <para>This section describes ...</para> + <title>Dharmesh's Mid-Summer Progress Updates</title> + <para>This section describes Dharmesh's progress on project "Integration project to deploy and use Mesos on a CloudStack based cloud"</para> + + <section id="dharmesh-introduction"> + <title>Introduction</title> + <para> + I am lagging a little in my timeline of the project. After the community bonding period, I have explored several things. My mentor, Sebastian has been really helpful and along with several others from the community. Along with my GSoC project I took up the task of resolving CLOUDSTACK-212 and it has been a wonderful experience. I am putting my best effort to complete the mesos integration as described in my proposal. + </para> + </section> + + <section id="cloudstack-212"> + <title>CLOUDSTACK-212 "Switch java package structure from com.cloud to org.apache"</title> + <para> + CLOOUDSTACK-212(https://issues.apache.org/jira/browse/CLOUDSTACK-212) is about migrating old com.cloud package structure to new org.apache to reflect the project move to Apache Software Foundation. + </para> + <para> + Rohit had taken the initiative and had already refactored cloud-api project to new package. When I looked at this bug, I thought it was a pretty straight forward task. I was not quite correct. + </para> + <para> + I used eclipse's refactoring capabilities for most of the refactoring. I used context-menu->refactor->rename with options of update - "references", "variable/method names" and "textual references" check-boxes checked. Also I disabled autobuild option as suggested. Also I disabled the CVS plugins as suggested by eclipse community the indexing by plugin while long refactoring was interfering and left garbled code. Even after these precautions, I noticed that eclipse was messing up some of the imports and especially bean-names in xml files. After correcting them manually, I got many test case failures. Upon investigation, I came to know that the error was because of resource folders of test cases. In short, I learned a lot. + </para> + <para> + Due to active development on master branch even between I create master-rebased-patch and apply-test-submit and one of the committer checks the applicability of the patch, the patch was failing due to new merges during this time. After several such attempt cycles, it became clear that this is not a good idea. + So after discussion with senior members of community, separate branch "namespacechanges" was created and I applied all the code refactoring there. Then one of the committer, Dave will cherry-pick them to master freezing other merge. I have submitted the patch as planned on 19th and it is currently being reviewed. + </para> + <para> + One of the great advantage of working on this bug was I got much better understanding of the cloudstack codebase. Also my understanding of unit testing with maven has become much more clearer. + </para> + </section> + + <section id="mesos-integration"> + <title>Mesos integration with cloudstack</title> + <para>There are multiple ways of implementing the project. I have explored following options with specific pros and cons.</para> + + + <section id="mesos-script"> + <title>Shell script to boot and configure mesos</title> + <para>This idea is to write a shell script to automate all the steps involved in running mesos over cloudstack. This is very flexible option as we have full power of shell.</para> + <itemizedlist> + <listitem> + <para>create security groups for master, slave and zookeeper.</para> + </listitem> + <listitem> + <para>get latest AMI number and get the image.</para> + </listitem> + <listitem> + <para>create device mapping</para> + </listitem> + <listitem> + <para>launch slave</para> + </listitem> + <listitem> + <para>launch master</para> + </listitem> + <listitem> + <para>launch zookeeper</para> + </listitem> + <listitem> + <para>wait for instances to come up</para> + </listitem> + <listitem> + <para>ssh-copy-ids</para> + </listitem> + <listitem> + <para>rsync</para> + </listitem> + <listitem> + <para>run mesos setup script</para> + </listitem> + </itemizedlist> + + <para>Since there exists a shell script within mesos codebase to create and configure mesos cluster on AWS, the idea is to use the same script and make use of cloudstack-aws API. Currently I am testing this script. + Following are the steps:</para> + <itemizedlist> + <listitem> + <para>enable aws-api on cloudstack.</para> + </listitem> + <listitem> + <para>create AMI or template with required dependencies.</para> + </listitem> + <listitem> + <para>download mesos.</para> + </listitem> + <listitem> + <para>configure boto environment to use with cloudstack</para> + </listitem> + <listitem> + <para>run mesos-aws script.</para> + </listitem> + </itemizedlist> + + <para>Pros: + <itemizedlist> + <listitem><para>Since the script is part of mesos codebase, it will be updated to work in future as well.</para></listitem> + </itemizedlist> + </para> + + </section> + + <section id="mesos-whirr"> + <title>WHIRR-121 "Creating Whirr service for mesos"</title> + <para>Whirr provides a comman API to deploy services to various clouds. Currently, it is highly hadoop centric. Tom white had done some work in Whirr community, but has not been updated for quite a long time.</para> + + <para>Pros: + <itemizedlist> + <listitem><para>Leverage Whirr API and tools.</para></listitem> + </itemizedlist> + </para> + + <para>Cons: + <itemizedlist> + <listitem><para>Dependence on yet another tool.</para></listitem> + </itemizedlist> + </para> + </section > + + <section id="mesos-cloudformation"> + <title>Creating a cloudformation template for mesos</title> + <para>The idea is to use AWS cloudformation APIs/functions, so that it can be used with any cloudformation tools. Within cloudstack, Stackmate project is implementing cloudformation service.</para> + + <para>Pros: + <itemizedlist> + <listitem><para>Leverage all the available tools for AWS cloudformation and stackmate</para></listitem> + </itemizedlist> + <itemizedlist> + <listitem><para>Potentially can be used on multiple clouds.</para></listitem> + </itemizedlist> + </para> + + <para>Cons: + <itemizedlist> + <listitem><para>Have to stay in the limits of ASW cloudformation API and otherwise have to use user-data to pass "shell commands", which will be not a maintainable solution in long term.</para></listitem> + </itemizedlist> + </para> + </section> + + </section> + + <section id="dharmesh-conclusion"> + <title>Conclusion</title> + <para> + I am very happy with the kind of things I have learned so far with the project. This includes: + </para> + <itemizedlist> + <listitem> + <para>Advanced git commands</para> + </listitem> + <listitem> + <para>Exposed to very large code base</para> + </listitem> + <listitem> + <para>Hidden features, methods and bugs of eclipse that will be useful refactoring large projects</para> + </listitem> + <listitem> + <para>How Unit testing work, especially with mvn</para> + </listitem> + <listitem> + <para>How to evaluate pros and cons of multiple options to achieve same functionality</para> + </listitem> + <listitem> + <para>Writing a blog</para> + </listitem> + </itemizedlist> + <para> + The experience gained from this project is invaluable and it is great that the Google Summer Of Code program exist. + </para> + </section> </section>
