[REEF-90]: Migrate REEF website code from a private repo to REEF repo This addressed the issue by
* Modifying the pom.xml file of $REEF_HOME to exclude README.* files in all directories when using `mvn apache-rat:check` * Adding a directory labelled `website` which contains the source code for the REEF website * Adding license headers to all website/src/* files except README.md * Adding the parent artifact and groupID and artifactID for REEF inside the website project's pom.xml file JIRA: [REEF-90] https://issues.apache.org/jira/browse/REEF-90 Pull Request: This closes #66 Project: http://git-wip-us.apache.org/repos/asf/incubator-reef/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-reef/commit/fa77cc63 Tree: http://git-wip-us.apache.org/repos/asf/incubator-reef/tree/fa77cc63 Diff: http://git-wip-us.apache.org/repos/asf/incubator-reef/diff/fa77cc63 Branch: refs/heads/master Commit: fa77cc63c530d878da7ab07b5e55962c720f9a40 Parents: 9defe61 Author: Josh Seol <[email protected]> Authored: Wed Feb 4 14:25:19 2015 +0900 Committer: Markus Weimer <[email protected]> Committed: Wed Feb 4 16:54:53 2015 -0800 ---------------------------------------------------------------------- pom.xml | 2 +- website/README.md | 24 + website/pom.xml | 57 +++ website/src/site/apt/team.apt | 88 ++++ website/src/site/markdown/api.md | 21 + website/src/site/markdown/coding-guideline.md | 51 +++ website/src/site/markdown/committer-guide.md | 159 +++++++ website/src/site/markdown/contributing.md | 159 +++++++ website/src/site/markdown/downloads.md | 75 +++ website/src/site/markdown/faq.md | 61 +++ website/src/site/markdown/glossary.md | 120 +++++ website/src/site/markdown/index.md | 55 +++ website/src/site/markdown/introduction.md | 63 +++ website/src/site/markdown/issue-tracker.md | 21 + website/src/site/markdown/license.md | 221 +++++++++ website/src/site/markdown/mailing-list.md | 40 ++ website/src/site/markdown/powered-by.md | 21 + website/src/site/markdown/reef-examples.md | 135 ++++++ website/src/site/markdown/tang.md | 455 +++++++++++++++++++ website/src/site/markdown/tutorial.md | 137 ++++++ website/src/site/markdown/wake.md | 109 +++++ website/src/site/resources/ApacheIncubator.png | Bin 0 -> 8626 bytes website/src/site/resources/REEFDiagram.png | Bin 0 -> 36456 bytes website/src/site/resources/REEFLogo.png | Bin 0 -> 1599 bytes .../src/site/resources/reef-architecture.png | Bin 0 -> 23411 bytes .../src/site/resources/states-horizontal.png | Bin 0 -> 27111 bytes website/src/site/site.xml | 122 +++++ 27 files changed, 2195 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index d24ece6..a334f75 100644 --- a/pom.xml +++ b/pom.xml @@ -239,7 +239,7 @@ under the License. <exclude>lang/java/.idea/**</exclude> <exclude>**/*.iml</exclude> <exclude>**/target/**</exclude> - <exclude>README.*</exclude> + <exclude>**/README.*</exclude> <!-- The below are sometimes created during tests --> <exclude>REEF_LOCAL_RUNTIME/**</exclude> <!-- The Visual Studio build files --> http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/README.md ---------------------------------------------------------------------- diff --git a/website/README.md b/website/README.md new file mode 100644 index 0000000..52b4c43 --- /dev/null +++ b/website/README.md @@ -0,0 +1,24 @@ +Website +=========== + +##Instructions + +####Generating the HTML files + +Navigate to the {$REEF_HOME}/website directory then use the command: + + mvn clean site:site + +This will generate a directory named "target" and inside, there will be a directory named "site". The site folder contains all files required for the website and index.html is the default page shown when the domain is first accessed. + +####Generating the Javadocs files for REEF + +Navigate to {$REEF_HOME} then use the command: + + mvn javadoc:aggregate + +This will create a directory named "apidocs" which includes all the necessary Javadocs for REEF. + +####Combining the HTML files with Javadocs files + +The site.xml file automatically searches for the apidocs at `apidocs/{$REEF_VERSION}/index.html`. When copying the apidocs to the svn, it is crucial to use this exact path because multiple index.html's exist and they must be referenced without ambiguity. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/pom.xml ---------------------------------------------------------------------- diff --git a/website/pom.xml b/website/pom.xml new file mode 100644 index 0000000..455ba52 --- /dev/null +++ b/website/pom.xml @@ -0,0 +1,57 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <artifactId>website</artifactId> + <name>REEF Website</name> + <parent> + <groupId>org.apache.reef</groupId> + <artifactId>reef-project</artifactId> + <version>0.11.0-incubating-SNAPSHOT</version> + <relativePath>..</relativePath> + </parent> + + <build> + <sourceDirectory>src</sourceDirectory> + <plugins> + <plugin> + <artifactId>maven-compiler-plugin</artifactId> + <version>3.1</version> + <configuration> + <source>1.7</source> + <target>1.7</target> + </configuration> + </plugin> + <plugin> + <artifactId>maven-site-plugin</artifactId> + <version>3.3</version> + <configuration> + <skipDeploy>true</skipDeploy> + </configuration> + <dependencies> + <dependency> + <groupId>org.apache.maven.doxia</groupId> + <artifactId>doxia-module-markdown</artifactId> + <version>1.6</version> + </dependency> + </dependencies> + </plugin> + </plugins> + </build> +</project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/apt/team.apt ---------------------------------------------------------------------- diff --git a/website/src/site/apt/team.apt b/website/src/site/apt/team.apt new file mode 100644 index 0000000..a6e99e2 --- /dev/null +++ b/website/src/site/apt/team.apt @@ -0,0 +1,88 @@ + ------ + Team + ------ + ------ + +~~ Licensed to the Apache Software Foundation (ASF) under one +~~ or more contributor license agreements. See the NOTICE file +~~ distributed with this work for additional information +~~ regarding copyright ownership. The ASF licenses this file +~~ to you under the Apache License, Version 2.0 (the +~~ "License"); you may not use this file except in compliance +~~ with the License. You may obtain a copy of the License at +~~ +~~ http://www.apache.org/licenses/LICENSE-2.0 + +~~ Unless required by applicable law or agreed to in writing, +~~ software distributed under the License is distributed on an +~~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +~~ KIND, either express or implied. See the License for the +~~ specific language governing permissions and limitations +~~ under the License. + +Team + + A successful project requires many people to play many roles. Some members write code or documentation, while others are valuable as testers, submitting patches and suggestions. + + The team is comprised of Project Management Committee Members and Contributors. Members have direct access to the source of a project and actively evolve the code-base. Contributors improve the project through submission of patches and suggestions to the Members. The number of Contributors to the project is unbounded. Get involved today. All contributions to the project are greatly appreciated. + +* Mentors + +*------------*-------------------------* +<<ID>> | <<Full Name>> +*------------*-------------------------* + cdouglas | Chris Douglas +*------------*-------------------------* + mattmann | Chris Mattmann +*------------*-------------------------* + rgardler | Ross Gardler +*------------*-------------------------* + omalley | Owen O'Malley +*------------*-------------------------* + + +*Committers + +*------------*--------------------------*---------------------------* + <<ID>> || <<Full Name>> || <<Affiliation(s)>> | +*------------*--------------------------*---------------------------* + weimer | Markus Weimer | Microsoft | +*------------*--------------------------*---------------------------* + motus | Sergiy Matusevych | Microsoft | +*------------*--------------------------*---------------------------* + juliaw | Julia Wang | Microsoft | +*------------*--------------------------*---------------------------* + shravan | Shravan M Narayanamurthy | Microsoft | +*------------*--------------------------*---------------------------* + yingdachen | Yingda Chen | Microsoft | +*------------*--------------------------*---------------------------* + tmajest | Tony Majestro | Microsoft | +*------------*--------------------------*---------------------------* + sezgin | Beysim Sezgin | Microsoft | +*------------*--------------------------*---------------------------* + shulmanb | Boris Shulman | Microsoft | +*------------*--------------------------*---------------------------* + sears | Russell Sears | Purestorage | +*------------*--------------------------*---------------------------* + jrlee | Jung Ryong Lee | SK Telecom | +*------------*--------------------------*---------------------------* + jerryjung | You Sun Jung | SK Telecom | +*------------*--------------------------*---------------------------* + dongjoon | Dong Joon Hyun | SK Telecom | +*------------*--------------------------*---------------------------* + joshrosen | Josh Rosen | UC Berekely | +*------------*--------------------------*---------------------------* + tcondie | Tyson Condie | UCLA | +*------------*--------------------------*---------------------------* + bmyers | Brandon Myers | University of Washington | +*------------*--------------------------*---------------------------* + yunseong | Yunseong Lee | Seoul National University | +*------------*--------------------------*---------------------------* + taegeonum | Taegeon Um | Seoul National University | +*------------*--------------------------*---------------------------* + johnyangk | Youngseok Yang | Seoul National University | +*------------*--------------------------*---------------------------* + chobrian | Brian Cho | Seoul National University | +*------------*--------------------------*---------------------------* + bgchun | Byung-Gon Chun | Seoul National University | +*------------*--------------------------*---------------------------* \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/api.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/api.md b/website/src/site/markdown/api.md new file mode 100644 index 0000000..6b0f182 --- /dev/null +++ b/website/src/site/markdown/api.md @@ -0,0 +1,21 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#API + +- [0.10.0-incubating](apidocs/0.10.0-incubating/index.html) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/coding-guideline.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/coding-guideline.md b/website/src/site/markdown/coding-guideline.md new file mode 100644 index 0000000..55a6d02 --- /dev/null +++ b/website/src/site/markdown/coding-guideline.md @@ -0,0 +1,51 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Coding Guidelines + +###The Basics + +We largely follow [Hadoop coding guidelines](http://wiki.apache.org/hadoop/CodeReviewChecklist): + +- Use 2 (TWO) spaces to indent code. +- Use LF (Unix style) line endings. +- Do **not** use the `@author` Javadoc tag. (Be modest ! :-)) +- For the rest, follow the original [Sun Java coding conventions](http://www.oracle.com/technetwork/java/codeconvtoc-136057.html). +- If some file has different formatting, try to match the existing style. +- Use [CheckStyle](http://checkstyle.sourceforge.net/) to verify your coding style. +- Avoid mixing both style and functionality changes in one commit. +- Make sure all files have the needed [license header](http://www.apache.org/legal/src-headers.html). + +###Comments + +We require committers to add comments to generate proper javadoc pages. In addition to this requirement, we encourage committers to add comments (if needed) to make code easier to understand. + +- Class +- Interface +- Public constructor +- Public method +- Important fields +- Code part that captures complex logic + +###On immutability of why you should make everything final + +REEF favors immutable objects over mutable ones. When you browse the code base, you see that there is an enormous number of final and readonly modifiers used. The reason for this stems from the distributed and parallel nature of REEF: What cannot be changed doesn't have to be guarded with locks. Hence, immutability is an excellent tool to achieve both thread safety and performance. Following this line of thought, we arrive at the following guidelines: + +- Make all instance variables possible `final`. +- Use the [Java Concurrency in Practice annotations](http://jcip.net.s3-website-us-east-1.amazonaws.com/) for the instance variables that aren't `final`. +- When a class has more than 3 non-`final` instance variables, consider breaking the class in two to limit the lock complexity. http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/committer-guide.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/committer-guide.md b/website/src/site/markdown/committer-guide.md new file mode 100644 index 0000000..ce094d9 --- /dev/null +++ b/website/src/site/markdown/committer-guide.md @@ -0,0 +1,159 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Committer Guide + +This captures committer-specific information beyond what's covered in the [contributions guide](contributing.html). + +###Accepting Pull Requests on GitHub + +The ASF INFRA team maintains [a mirror of our git repository over on GitHub](https://github.com/apache/incubator-reef).That mirror is strictly one-way: changes from the Apache git get copied over, but not the other way around. Further, the mirror on GitHub is read-only for everyone. Which means we can neither accept nor close pull requests filled there via the GitHub web UI. However, we want to allow for contributions via GitHub Pull Requests. Here's how: + +####Add the ASF git repository + +If you have not done so already, add the ASF git repository. For example, to add as the "apache" remote repository: + + $ git remote add apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git + +As a result, you can refer to the ASF git as "apache". An example setup, with a the GitHub mirror at upstream and a forked GitHub origin at `{username}`: + + $ git remote -v + apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git (fetch) + apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git (push) + origin https://github.com/{username}/incubator-reef.git (fetch) + origin https://github.com/{username}/incubator-reef.git (push) + upstream https://github.com/apache/incubator-reef.git (fetch) + upstream https://github.com/apache/incubator-reef.git (push) + +####Merge changes + +The next step is to merge all changes on the Pull Request as a single commit. There are two methods here: (A) pull the branch from the GitHub Pull Request and squash commits, or (B) get the `.diff` from GitHub and manually create a commit from this information. Each option has its pros and cons. With method A, git does some of the tedious work of preserving commit messages; but in some cases the squash may not apply cleanly, and the merger will have to carefully resolve conflicts. With method B, the `.diff` will apply cleanly (given that the branch is up-to-date, i.e. the GitHub GUI states that the "This pull request can be automatically merged by project collaborators"); but the merger will have to carefully copy over commit messages and edit author information. + +####Commit Message + +Whichever method you choose, the following should be included in the final commit message: + +- Pull request description and commit comments +- A link to the JIRA this is addressing. +- The text "closes #PRNUMBER", where PRNUMBER is the number of the pull request, e.g. "10" + +Following the last statement will close the GitHub pull request. It is important to close via the commit message, because we cannot close pull requests via the GitHub Web UI. + +#####Example Commit message (80 columns) + + [REEF-33] Allow Tasks to initiate an Evaluator Heartbeat + This adds the class HeartBeatTriggerManager which can be used by a Task to + initiate an Evaluator Heartbeat. It is used by injecting it into the Task. + + JIRA: + [REEF-33] https://issues.apache.org/jira/browse/REEF-33 + + Pull Request: + Closes #24 + + Author: + AuthorName AuthorEmail + +####Method A + +#####A-1. Create a branch on your local git repository + +You want to make sure that that branch is current with the the master branch on Apache git: + + $ git checkout -b BRANCH_NAME + $ git pull apache master + +This assumes that you called the remote git repository at the ASF "apache". You can name the branch however you want, but it is customary to name them either after the pull request number or the JIRA id, e.g. REEF-24. + +#####A-2. Pull the contribution into that branch + +This is as simple as + + $ git pull GIT_REPOSITORY_OF_THE_CONTRIBUTOR BRANCH_NAME_OF_THE_CONTRIBUTION + +However, the GitHub web ui makes it somewhat hard to get these two strings. The email from GitHub that informs us of Pull Requests makes this really obvious, though. Consider this quote from pull [request #10](https://github.com/apache/incubator-reef/pull/10): + +>**You can merge this Pull Request by running** +>>`git pull https://github.com/jwang98052/incubator-reef juwang-logfactory` + +This copies the changes from the given remote branch into the one we just created locally. + +#####A-3. Check the pull request + +1. Make sure the code compiles and all tests pass. +2. If the code touches code that you suspect might break on YARN or Mesos, please test on those environments. If you don't have access to a test environment, ask on the mailing list for help. +3. Make sure the code adheres to our [coding guidelines](coding-guideline.html). +4. Make sure that the additions don't create errors in a [Apache RAT](http://creadur.apache.org/rat/) check via `mvn apache-rat:check` + +#####A-4. Rebase the branch onto current apache master + + # Make sure we have the latest bits + $ git fetch apache + # Rebase + $ git rebase -i apache/master + +In the rebase process, make sure that the contribution is squashed to a single commit. From the list of commits, "pick" the commit in the first line (the oldest), and "squash" the commits in the remaining lines: + + pick 7387a49 Comment for first commit + squash 3371411 Comment for second commit + squash 9bf956d Comment for third commit + +[Chapter 3.6](http://www.git-scm.com/book/en/v2/Git-Branching-Rebasing) and [Chapter 7.6](http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) of the [Git Book](http://www.git-scm.com/book/en/v2) contains lots of information on what this means. + +Please make sure that the commit message contains the information given in "Commit Message" above. The latest commit can be changed at any time with the command: + + $ git commit --amend + +#####A-5. Push the code into apache's git + +This is a good time to reflect back on this change and whether it is likely to break the build. If you are certain that it won't, go ahead and do: + + $ git checkout master + $ git merge BRANCH_NAME + $ git push apache master + +This pushes the current branch into the master branch hosted on Apache's git repository. From there, it will be mirrored onto GitHub. And by virtue of the comment added above to the commit message, GitHub will now close the Pull Request. + +####Method B + +#####B-1. Update local master + +In this method, you will work directly on your local master branch. Make sure you have the latest changes on your local master branch, by running: + + $ git pull apache master + +#####B-2. Download the .diff and apply it + +You can download the `.diff` file by appending `.diff` to the GitHub Pull Request url. This file will contain the exact changes that are shown in "Files changed" in the GitHub GUI. For example, for `https://github.com/apache/incubator-reef/pull/24` the `.diff` file is `https://github.com/apache/incubator-reef/pull/24.diff`. Using this url, run apply: + + $ wget https://github.com/apache/incubator-reef/pull/24.diff + $ git apply 24.diff + +#####B-3. Commit and edit author information + +Commit all files, making sure to include all modified and **new** files. In the commit message, make sure to include all information given in "Commit Message" above. After committing you must also change the author information to reflect the original author (and not yourself): + + $ git commit --amend --author "Original Author Name <[email protected]>" + +#####B-4. Push the code into apache's git + +Now push your single commit to apache git: + + $ git push apache master + +This pushes the current branch into the master branch hosted on Apache's git repository. From there, it will be mirrored onto GitHub. And by virtue of the comment added above to the commit message, GitHub will now close the Pull Request. http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/contributing.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/contributing.md b/website/src/site/markdown/contributing.md new file mode 100644 index 0000000..8e26456 --- /dev/null +++ b/website/src/site/markdown/contributing.md @@ -0,0 +1,159 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Contributing to REEF + +####First things first: Welcome! + +Arriving on this page means that you are interested in helping us out. For that: Welcome and thank you! REEF is a community driven project and we always welcome new people joining us. We are inviting contributions in many forms to all parts of the project, including: + +- Bug reports regarding the software, the documentation, the website, guides like this, etc. +- Graphics (for instance, [we don't have a logo yet](https://issues.apache.org/jira/browse/REEF-14)). +- Documentation updates, tutorials, examples. +- Code: Patches, new features, etc + +####Getting started: Join the community. + +The first step is to **join the community** by joining our [mailing list](mailing-list.html). This is where all discussions regarding the project are happening, where we answer each other's questions and where you will find a friendly bunch to welcome you. + +If you want to work on the REEF code base, it is a good idea to [learn how to compile and test REEF](tutorial.html). + +####Finding things to work on + +At any given time, there is any number of [open, unassigned issues on REEF](https://issues.apache.org/jira/issues/?jql=project%20%3D%20REEF%20AND%20status%20%3D%20Open%20AND%20resolution%20%3D%20Unresolved%20AND%20assignee%20in%20\(EMPTY\)%20ORDER%20BY%20priority%20DESC). Note that that list doesn't only contain coding tasks, but also documentation, graphics work, the website and so on. We use JIRA to manage all open todos for the project, software or otherwise. However, some of the items on that list might since have become obsolete or such. Hence, it is always a good idea to get in touch with the rest of the community on the mailing list before getting started. + +####Code contribution process + +While contributing to REEF, you will encounter ASF infrastructure, GitHub infrastructure, and follow ASF and REEF-specific customs. The most important of those customs is to communicate throughout the whole process. That way, the REEF community has a chance to help you along the way. + +####Before you start the work + +An important part of the process comes before you start the contribution process. Most issues should first be brought up in the [[email protected]](mailto:[email protected]) mailing list. If you haven't done so yet, [subscribe to the list](mailing-list.html). After discussion, you or one of the other developers will create an Issue on [ASF JIRA](https://issues.apache.org/jira/browse/REEF/). Again, create an account if you don't have one. Write a succinct description of the Issue, making sure to incorporate any discussion from the mailing list. + +And once you are ready to make changes, have a look at the [coding guidelines](coding-guideline.html) to make sure your code agrees with the rest of REEF. + +####Creating the contribution as a GitHub Pull Request + +REEF uses [GitHub's Pull Requests](https://help.github.com/articles/using-pull-requests/) as the main method to accept changes: you fork REEF into your own repository, make changes, and then send a pull request to the GitHub repository. In more detail: + +1. Fork repository +2. Create branch +3. Make changes in your local machine +4. Merge the master branch into your branch +5. Push commits into the your remote repo (forked) +6. Send a pull request. +7. Participate in the code review. + +#####1. Fork repository + +First, you need to fork the REEF repository. Go to the [Github repository](https://github.com/apache/incubator-reef) mirrored from ASF, and click "Fork" button. Then you will have your own repository. Clone this repository to your local machine: + + $ git clone https://github.com/{your_alias}/incubator-reef.git + +Then, add the Apache GitHub repository as upstream: + + $ git remote add upstream https://github.com/apache/incubator-reef + +A correct git configuration should look similar to the following, with an origin and upstream: + + $ git remote -v + origin https://github.com/{your_alias}/incubator-reef.git (fetch) + origin https://github.com/{your_alias}/incubator-reef.git (push) + upstream https://github.com/apache/incubator-reef.git (fetch) + upstream https://github.com/apache/incubator-reef.git (push) + +If you have an `apache.org` email address, now is the time to [configure git](https://git-wip-us.apache.org/) to use it: + + $ git config user.name "My Name Here" + $ git config user.email [email protected] + +#####2. Create a branch to work on +Before making changes, you have to make sure the issue to resolve (e.g. fix a bug, implement a new feature, etc) is registered in the [REEF JIRA](https://issues.apache.org/jira/browse/REEF). Create a branch to address the issue on your own. The name of the branch should reference the issue, e.g., `REEF-{issue_number}`. You can take a look how others name their branches. + +#####3. Make changes in your local machine +Write the code and make commits as usual. Make sure all new files contain the [ASF licensing header](https://github.com/apache/incubator-reef/blob/master/LICENSE_HEADER.txt). + +#####4. Merge the master branch into your branch +Before sending a pull request, you should make sure that your branch includes the latest changes in the master branch. Please run the following: + + $ git fetch upstream + $ git checkout {your_branch} # Skip this step if you are already on your branch. + $ git merge upstream/master + +Resolve the conflicts if they exist. Test with the merged code so that it does not break the system. Then, check that Apache headers are in place where needed, by running RAT: + + $ mvn apache-rat:check + +Finally, as a courtesy to the merger, you can rebase to master and squash all the commits from your PR into one: + + # Rebase + $ git rebase -i upstream/master + +In the rebase process, make sure that the contribution is squashed to a single commit. From the list of commits, "pick" the commit in the first line (the oldest), and "squash" the commits in the remaining lines: + + pick  7387a49 Comment for first commit + squash 3371411 Comment for second commit + squash 9bf956d Comment for third commit + +[Chapter 3.6](http://www.git-scm.com/book/en/v2/Git-Branching-Rebasing) and [Chapter 7.6](http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) of the [Git Book](http://www.git-scm.com/book/en/v2) contains lots of information on what this means. + +In this process, git allows you to edit the commit message for the final, squashed commit. This commit message will serve as the description of the pull request and will in all likelihood appear in verbatim in the REEF history. In other words: Spend some time making it good. A common template for this commit message is: + + [REEF-JIRA_ISSUE_NUMBER]: THE_TITLE_OF_THE_JIRA + + This addressed the issue by + * INSERT_DESCRIPTION_OF_SOMETHING + * INSERT_DESCRIPTION_OF_SOMETHING_ELSE + * ... + + JIRA: [REEF-JIRA_ISSUE_NUMBER](https://issues.apache.org/jira/browse/REEF-JIRA_ISSUE_NUMBER) + +You can get a good idea how other people write their commit messages by inspecting the output of `git log`. + +#####5. Push commits into the your remote repo (forked) + +You're almost done! Push your commits into your own repo. + + $ git push origin HEAD + +#####6. Send a pull request + +It is time to send a pull request. If you do not know much about pull requests, you may want to read this [article](https://help.github.com/articles/using-pull-requests/). + +If you go to the repository at the Github website, you can notice that "Compare & Pull request" button appears. Click the button or move into "pull request" tab if you cannot find the button. + +When you create a pull request, you choose the branches to compare and merge. Choose the base as `apache:master` and the head `{your_alias}:{branch_name}`. The description will be the message of your one commit. Feel free to edit it if you aren't satisfied with your commit message. + +Please **update the JIRA issue with a link to the Pull Request**. This triggers an email to the mailing list, which will hopefully result in a committer stepping up for a code review. + +#####7. The code review + +REEF follows a review-then-commit (RTC) model. We perform all our code reviews on GitHub: Community members will leave comments on your pull request and suggest changes. You can have [a look at prior pull requests](https://github.com/apache/incubator-reef/pulls?q=is%3Apr+is%3Aclosed) to get an idea of what to expect. During this process, you may want to change your code. You can do that by pushing additional commits to your branch. Don't worry about squashing the commits into one at this stage. This will be done by the committer once your code is merged into REEF. + +When the code review concludes, one of Committers will merge your work into the REEF codebase. Good job! + +#####8. Merge the pull request (Committers only) + +If you are a committer, you can follow the steps in the [Committer Guide](committer-guide.html) to merge the pull request. Of course, you won't be merging your own pull requests. Nudge committers as needed to make sure everything gets merged correctly. + +####Other guides + +There are other pages in this area you might be interested in: + +- [Coding Guidelines](coding-guideline.html) +- [REEF Tutorial](tutorial.html) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/downloads.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/downloads.md b/website/src/site/markdown/downloads.md new file mode 100644 index 0000000..1a8cf59 --- /dev/null +++ b/website/src/site/markdown/downloads.md @@ -0,0 +1,75 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Downloads + +##Releases + +- [0.10.0-incubating](#0.10.0-incubating) + +###<a name="0.10.0-incubating"></a>0.10.0-incubating + +#####Downloads + +- [apache-reef-0.10.0-incubating.tar.gz](http://www.apache.org/dist/incubator/reef/0.10.0-incubating/apache-reef-0.10.0-incubating.tar.gz) +- [apache-reef-0.10.0-incubating.tar.gz.asc](http://www.apache.org/dist/incubator/reef/0.10.0-incubating/apache-reef-0.10.0-incubating.tar.gz.asc) +- [apache-reef-0.10.0-incubating.tar.gz.md5](http://www.apache.org/dist/incubator/reef/0.10.0-incubating/apache-reef-0.10.0-incubating.tar.gz.md5) +- [apache-reef-0.10.0-incubating.tar.gz.sha1](http://www.apache.org/dist/incubator/reef/0.10.0-incubating/apache-reef-0.10.0-incubating.tar.gz.sha1) +- [apache-reef-0.10.0-incubating.tar.gz.sha512](http://www.apache.org/dist/incubator/reef/0.10.0-incubating/apache-reef-0.10.0-incubating.tar.gz.sha512) + +##Development and Maintenance Branches + +If you are interested in working with the newest under-development code or contributing to REEF, you can also check out the master branch from Git: + + $ git clone git://git.apache.org/incubator-reef.git + +##How to verify the integrity of the files + +It is essential that you verify the integrity of the downloaded files using the PGP or MD5 signatures. Please read [Verifying Apache HTTP Server Releases](http://www.apache.org/info/verification.html) for more information on why you should verify our releases. + +The PGP signatures can be verified using [PGP](http://www.pgpi.org/) or [GPG](https://www.gnupg.org/). First download the [KEYS](http://www.apache.org/dist/incubator/reef/KEYS) as well as the `*.asc` signature file for the relevant distribution. Make sure you get these files from the [main distribution directory](http://www.apache.org/dist/incubator/reef/) rather than from a mirror. Then verify the signatures using one of the following sets of commands: + + % pgpk -a KEYS + % pgpv downloaded_file.asc + +or + + % pgp -ka KEYS + % pgp downloaded_file.asc + +or + + % gpg --import KEYS + % gpg --verify downloaded_file.asc + +Alternatively, you can verify the MD5 signature on the files. A Unix/Linux program called md5 or md5sum is included in most distributions. It is also available as part of [GNU Textutils](http://www.gnu.org/software/textutils/textutils.html). Windows users can get binary md5 programs from these (and likely other) places: + + +- http://www.md5summer.org/ +- http://www.fourmilab.ch/md5/ +- http://www.pc-tools.net/win32/md5sums/ + +##Maven Dependencies + +REEF artifacts are hosted in [Maven Central](http://search.maven.org/#search|ga|1|org.apache.reef) and can be added to your Maven project using the following format: + + <dependency> + <groupId>org.apache.reef</groupId> + <artifactId>reef-project</artifactId> + <version>{$REEF_VERSION}</version> + </dependency> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/faq.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/faq.md b/website/src/site/markdown/faq.md new file mode 100644 index 0000000..71b24a8 --- /dev/null +++ b/website/src/site/markdown/faq.md @@ -0,0 +1,61 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#FAQ + +1. [Who is REEF for?](#who) +2. [Why did it come about?](#why) +3. [What is Tang?](#tang) +4. [What is Wake?](#wake) +5. [How can I get started?](#how) + +###<a name="who"></a>Who is REEF for? + +REEF is for developers of data processing systems on cloud computing platforms that provide fine-grained resource allocations. REEF provides system authors with a centralized (pluggable) control flow that embeds a user-defined system controller called the Job Driver. The interfaces associated with the Job Driver are event driven; events signal resource allocations and failures, various states associated with task executions and communication channels, alarms based on wall-clock or logical time, and so on. REEF also aims to package a variety of data-processing libraries (e.g., high-bandwidth shuffle, relational operators, low-latency group communication, etc.) in a reusable form. Authors of big data systems and toolkits can leverage REEF to immediately begin development of their application specific data flow, while reusing packaged libraries where they make sense. + +___________________________________________________________________________ + +###<a name="why"></a>Why did it come about? + +Traditional data-processing systems are built around a single programming model (like SQL or MapReduce) and a runtime (query) engine. These systems assume full ownership over the machine resources used to execute compiled queries. For example, Hadoop (version one) supports the MapReduce programming model, which is used to express jobs that execute a map step followed by an optional reduce step. Each step is carried out by some number of parallel tasks. The Hadoop runtime is built on a single master (the JobTracker) that schedules map and reduce tasks on a set of workers (TaskTrackers) that expose fixed-sized task "slots". This design leads to three key problems in Hadoop: + + +1. The resources tied to a TaskTracker are provisioned for MapReduce only. +2. Clients must speak some form of MapReduce in order to make use of cluster resources, and in turn, gain compute access to the data that lives there. +3. Poor cluster utilization, especially in the case of idle resources (slots) due to straggler tasks. + +With YARN (Hadoop version two), resource management has been decoupled from the MapReduce programming model in Hadoop, freeing cluster resources from slotted formats, and opening the door to programming frameworks beyond MapReduce. It is well understood that while enticingly simple and fault-tolerant, the MapReduce model is not ideal for many applications, especially iterative or recursive workloads like machine learning and graph processing, and those that tremendously benefit from main memory (as opposed to disk based) computation. A variety of big data systems stem from this insight: Microsoft's Dryad, Apache Spark, Google's Pregel, CMU's GraphLab and UCI's AsterixDB, to name a few. Each of these systems add unique capabilities, but form islands around key functionalities, making it hard to share both data and compute resources between them. YARN, and related resource managers, move us one step closer toward a unified Big Data system stack. The goal of REEF is to provide the next level of detail in this layering. + +______________________________________________________ + +###<a name="tang"></a>What is Tang? + +Tang is a dependency injection Framework co-developed with REEF. It has extensive documentation which can be found [here](tang.html). + +________________________________________________ + +###<a name="wake"></a>What is Wake? + +Please refer to [this](wake.html) section. + +________________________________________________ + + +###<a name="how"></a>How can I get started? + +Check out the [REEF Tutorial](tutorial.html) and the [Contributing](contributing.html) page and join the community! \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/glossary.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/glossary.md b/website/src/site/markdown/glossary.md new file mode 100644 index 0000000..d424d71 --- /dev/null +++ b/website/src/site/markdown/glossary.md @@ -0,0 +1,120 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Glossary + +- [Context](#context) +- [Driver](#driver) +- [Evaluator](#evaluator) +- [Task](#task) + +###<a name="context"></a>Context + +Contexts are a way to structure the state and Configuration of an Evaluator. A Context exists on one and only one individual Evaluator. Each Evaluator has at least one Context, which we refer to as the *root* Context. This root context is special, as it is *required* on all Evaluators and because closing it is synonymous to releasing the Evaluator. In many simple REEF programs, this is the only Context used and it is therefore convenient to think of it as synonymous to "the Evaluator": The Driver can submit Tasks to it, is notifified when they are done and can close the root context when it wants to dispose of the Evaluator. + +Contexts are formed by calls to `submitContext()` to the event types that allow this (`AllocatedEvaluator` and `ActiveContext`) Contexts are the main way for an Evaluator to be exposed and accessed. For instance, Tasks are submitted to an `ActiveContext` which represents the top Context on the Evaluator. + +Beyond this, a Driver can submit a Context to the root, or in fact any, Context, as long as the resulting structure is that of a stack: The root Context forms the bottom of the stack, the top-most Context is called *active*, hence the `ActiveContext` event. The two can be one and the same, and often are: The root Context is the subject of the first `ActiveContext` event on an Evaluator. + +Nomenclature: When Context B is submitted to an already existing Context A, we say that Context A is the parent Context of Context B. Also, Context B is the child of Context A. + +It is only the `ActiveContext` that allows the submission of Tasks or child Contexts. Hence, one can think of the whole Evaluator structure as that of a stack: the root Context at the bottom, layers of Contexts in the middle and either the current `ActiveContext` or the current Task at the top. + +####Objects and Configuration: What's in a Context? + +It is convenient to think of a Context as a `Configuration` that gets merged with the `Configuration` supplied for Tasks and child Contexts. While not entirely true (see below), this view allows us to show just *why* Contexts are a convenient construct. + +It is often the case that subsequent tasks that get executed on an Evaluator want access to the same Configuration variables and / or the same objects. Consider a simple `LinkedList` bound to a named parameter. If that linked list is part of the subsequent Task `Configurations` submited, each Task will get its very *own* `LinkedList`. If the named parameter is bound in the Context `Configuration`, all Tasks subsequently submitted to the Context will get the very *same* `LinkedList` instance. + +####Contexts are (Tang) Injectors + +This mechanism is implemented by using Tang's `Injector`s. On the Evaluator, a Task is launched by first *forking* the Context's `Injector` with the Task`Configuration` and then requesting an instance of the `Task` interface from that forked `Injector`. By this mechanism and the fact that objects are singletons with respect to an `Injector` in Tang, object sharing can be implemented. All objects already instantiated on the Context `Injector` will also be referenced by the Task`Injector`. Hence, the `LinkedList` in the example above would be shared amongst subsequent Task `Injectors` in the construction of the `Task` instance. + +###<a name="driver"></a>Driver + +REEF imposes a centralized control flow design on applications: All events are routed to the master node, called the Driver. REEF also prescribes event-driven programming for the Driver. In that sense, the application provided Driver is a collection of event handlers for the various events exposed in `DriverConfiguration`. While most of these deal with occurrences during the computation (Evaluator allocation, Task launch, ...), several stand out as life-cycle events of the Driver, and therefore the application: + +####ON_START + +This event is triggered by REEF when the Driver is ready to start executing. At this point communication with the Resource Manager has been established, all event handlers have been instantiated and the event graph in the Driver was deemed to be complete enough to start. In a typical application, this is when the Driver requests the first set of Evaluators. + +####ON_STOP + +This event is fired right before the Driver shuts down. REEF determines Driver shutdown by proof that no future events can happen: + +- The Clock has no outstanding alarms. +- The resource manager has no outstanding requests. +- The application has no Evaluators allocated anymore. + +Hence, the `ON_STOP` event can be used to prevent Driver shutdown, e.g. in applications that need to wait for external triggers (e.g. a UI) to proceed. To do so, one would schedule an alarm in the `ON_STOP` handler. + +###<a name="evaluator"></a>Evaluator + +####Evaluators and Tasks + +The Evaluator is the runtime environment for Tasks. On one Evaluator, there is either no or one Task executing at any given point in time. Different or multiple executions of the same Tasks can be executed in sequence on an Evaluator. The Evaluator and Task lifecycle are decoupled: Whenever a Task finishes, the Driver receives the CompletedTask event, which contains a reference to the Evaluator the Task executed on. It is then up to the Driver to decide whether to return the Evaluator to the resource manager or to make other use of it, e.g. by submitting another task. + +####Evaluators and Contexts + +Contexts are REEF's form of state management inside of the Evaluator. See the [Context](#context) section for more information. + +####Evaluators and the Resource Manager + +On typical resource managers, an Evaluator is a process executing inside a container. Depending on the resource manager, that process may or may not be guarded by a resource or security isolation layer. + +This also means that the Evaluator, not the Task, is the unit of resource consumption: while an Evaluator is occupying a Container, that Container is "used" from the perspective of the Resource Manager.That is true even if the Evaluator is idle from the perspective of the Driver, i.e. when no Task is running on it. + +###<a name="task"></a>Task + +####Definition + +A Task in REEF is a unit of work to be executed on an Evaluator. In its simplest form, a Task is merely an object implementing the Task interface which prescribes a single method: + + public byte[] call(byte[] input); + +From REEF's perspective, a Task is therefore a single threaded method call. It starts when entering the call method. It is a `RunningTask` while it hasn't returned from it and is a `CompletedTask` when it has. Should there be an Exception thrown by `call()`, we call it a `FailedTask`. + +Task identity is established by a user-defined string set in `TaskConfiguration.IDENTIFIER`. All subsequent task-related events in the Driver will carry that ID. Note that REEF doesn't take any particular precautions to ensure unique Task identifiers. It is up to the application to do so. While technically feasible to assign the same identifier to multiple Tasks, this isn't advised as it makes error handling, debugging and logging unnecessarily hard. + +####Inputs and outputs of a Task + +The return value of the `call` method will be made available to the Driver as part of the `CompletedTask` event. Note that it isn't advised to return large values in this fashion, but merely small control flow or status information. Sending large data on this channel creates the risk of overloading the Driver at scale. The networking APIs provided by REEF IO are much better suited for data transmissions than this channel. + +The parameter given to the call method is also to be used in a similar fashion: The Driver passes its value as part of the Task submission. It is meant e.g. to convey a restart point for the task. Note that the same functionality can now be better provided by Tang and a constructor parameter. + +####Communicating between a Task and a Driver + +REEF provides some facilities to communicate between a Driver and a Task. These mostly stem from allowing application code to "free-ride" on REEF's control flow channels such as the heartbeat between the Evaluator and the Task. + +#####Sending a message from the Driver to a Task + +REEF maintains a heartbeat between any Evaluator and the Driver. There are two ways by which a heartbeat can be triggered. + +- Upon some schedule (which may also vary at runtime due to load conditions on the Driver), each Evaluator will report its current status to the Driver. This is used by the Driver to maintain health status and load statistics of the Evaluators. + +- Whenever the status of the Evaluator changes (e.g. when a Task completes), a Heartbeat is triggered immediately. + +Whenever the Evaluator performs a heartbeat, it will ask the Task whether it has any message to share with the Driver by inquiring the class registered in `TaskConfiguration.ON_SEND_MESSAGE`. It is wise for that message to be small, as we otherwise run the risk of overwhelming the Driver with heartbeat traffic at scale. + +####Multithreaded Tasks + +Just because REEF views a Task as a method call doesn't restrict the Task to be single threaded. A Task is free to spawn threads in the course of its execution. However, a Task that does so needs to take care of a few considerations: + +- All Threads spawned need to exit before the `Task.call()` method returns. Otherwise, you run the risk of resource leakage. + +- Exceptions on spawned Threads need to be caught and re-thrown by the `Thread.call()` method. Before that, all spawned threads need to be shut down, just like during a normal exit of `Task.call()`. If an exception from an another thread isn't caught, REEF's JVM level exception handler will catch it and declare a FailedEvaluator. This is inefficient, but not technically wrong: The Driver will then have to allocate another Evaluator and try again. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/index.md b/website/src/site/markdown/index.md new file mode 100644 index 0000000..3c5a05e --- /dev/null +++ b/website/src/site/markdown/index.md @@ -0,0 +1,55 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Apache REEF + +###What is REEF? + +**REEF**, the Retainable Evaluator Execution Framework, is our approach to simplify and unify the lower layers of big data systems on modern resource managers. + +For managers like Apache YARN, Apache Mesos, Google Omega, and Facebook Corona, REEF provides a centralized control plane abstraction that can be used to build a decentralized data plane for supporting big data systems. Special consideration is given to graph computation and machine learning applications, both of which require data *retention* on allocated resources to execute multiple passes over the data. + +More broadly, applications that run on YARN will have the need for a variety of data-processing tasks e.g., data shuffle, group communication, aggregation, checkpointing, and many more. Rather than reimplement these for each application, REEF aims to provide them in a library form, so that they can be reused by higher-level applications and tuned for a specific domain problem e.g., Machine Learning. + +In that sense, our long-term vision is that REEF will mature into a Big Data Application Server, that will host a variety of tool kits and applications, on modern resource managers. + +<div style="text-align:center" markdown="1"> + <img> + <img src ="REEFDiagram.png"/> + </img> +</div> + +###How can I get started? + +The official home for the REEF (and Tang and Wake) source code is at the Apache Software Foundation. You can check out the current code via: + + $ git clone git://git.apache.org/incubator-reef.git + +or directly access its GitHub page [here](https://github.com/apache/incubator-reef). + +Detailed information about REEF and using it can be found in the [FAQ](faq.html) and the [Tutorial](tutorial.html). + +If you wish to contribute, start at the [Contributing](contributing.html) tutorial or the [Committer Guide](committer-guide.html)! + +###Further questions? + +Please visit our [Frequently Asked Questions](faq.html) page or use our [Mailing List](mailing-list.html) to send us a question! + +###Disclaimer + +Apache REEF is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/introduction.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/introduction.md b/website/src/site/markdown/introduction.md new file mode 100644 index 0000000..37def4d --- /dev/null +++ b/website/src/site/markdown/introduction.md @@ -0,0 +1,63 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Introduction to REEF + +It is instructive to first remind us of how MapReduce works: Hadoop Map-Reduce schedules compute tasks on containers for executing map and reduce functions on record data. The basic structure of a Map-Reduce job is as follows. For each input block, schedule a map task that passes each internal record to a user-defined map function and materializes the output in key-sorted order. Further, assign a user-defined number of reduce tasks to non-overlapping portions of the key-space from the map output, then shuffle it across the network to where the respective reduce task is scheduled. For each reduce task, perform a global key-based sort on the shuffled data, group it by key and call the reduce function on each record group; storing the output in a durable fashion (i.e., HDFS). + +From the perspective of the scheduler, a number of issues arise that must be appropriately handled in order to scale-out to massive datasets. First, each map task should be scheduled close to where the input block resides; ideally on the same machine or rack. Second, failures can occur at the task level at any step; requiring backup tasks to be scheduled or the job being aborted. Third, performance bottlenecks can cause an imbalance in the task-level progress. The scheduler must react to these stragglers by scheduling clones and incorporating the logical task that crosses the finish line first. + +Anyone of these issues can limit the scale-out degree of a Map-Reduce job. In what follows, we perscribe a scheduler framework that provides task life-cycle management mechanisms. Using this framework, we developed a complete version of the Map-Reduce runtime that addresses the above issues. Our framework is designed around three components. + +1. `Driver`: captures the client code embodying the scheduler +2. `Evaluator`: provides a runtime environment on a container. +3. `Task`: encapsulates the task-level client code to be executed in an Evaluator. + +Below, we describe the client facing interfaces to these components. The core REEF control flow design is based on the reactive extensions (Rx), which enforce asynchronous message-passing method signatures. In Rx terms, interfaces are based on an observer pattern, which expose methods that accept messages from a (possibly remote) asynchronous caller. + +<br></br> + +<div style="text-align:center" markdown="1"> + <img src="reef-architecture.png"></img> +</div> + +*A running REEF job with two `RunningEvaluators` and one `RunningTask`* + +<br></br> + +The above figure presents the REEF components in terms of a running application, which is written in terms of a `Driver` and task-specific `Task` modules. The application code is packaged and submitted to a REEF client API, which in turn submits a REEF-AM configuration to YARN. The REEF-AM contains a runtime for launching the `Driver` and client libraries for requesting Evaluators and launching `Activities`. When a request for `Evaluators` is made, the REEF-AM negotiates containers with the YARN-RM and launches an `Evaluator` runtime on the YARN-NM that hosts the allocated container. In turn, the `Driver` is given an `Evaluator` object reference, which it uses to submit an `Task`. The `Driver` is also given a `Task` object reference, which it may use to send messages to the `Task` running in the `Evaluator`. The REEF layer implements these communication channels and encodes the computational life-cycle as state transitions, which are surfaced to the `Driver` in the form of Rx messag es. + +###Computational Life-Cycle + +<br></br> + +<div style="text-align:center" markdown="1"> + <img src="states-horizontal.png"></img> +</div> + +*States of `Evaluator`, `Contexts`, and `Activities`* + +<br></br> + +The Figure above describes the state transitions for (a) `Evaluator` and `Context` and (b) `Task` components. Each state transition is associated with an object reference that is surfaced to the `Driver` in an Rx-style interface. For instance, when the YARN-RM notifies the REEF-AM of an allocated container, the `Driver` is given an `AllocatedEvaluator` object; containing methods for adding configurations (i.e., for data services, see below) and file resources, and submit methods that bootstraps the `Evaluator` runtime on the YARN-NM. When an `Evaluator` bootstrap successfully completes, the `Driver` is given an `ActiveContext` object, which can be used to launch `Activities` or to initiate a close, which triggers a shutdown event at the `Evaluator` runtime and a subsequent container release at the YARN-RM. If at any point a failure occurs, the `Driver` is passed a `FailedEvaluator` object; containing an exception trace when possible. + +Recall that the `Driver` launches a `Task` on a submit method call from the `ActiveContext` reference. This state transition is denoted in the Figure above by the edge labeled submit; spanning the two state machines. The REEF-AM passes a `RunningActivity` object to the `Driver` after receiving confirmation of a successful `Task` start or resume. The `Driver` may use the `RunningActivity` reference to close or suspend the execution; triggering a `CompletedActivity` or `SuspendedActivity` object reference to the `Driver`. The `SuspendedActivity` object contains a memento used to resume the execution on some (possibly alternative) `ActiveContext`. Exceptions during the `Task` execution are surfaced to the `Driver` in the form of a `FailedActivity`, which contains the actual exception object. + +###Task Component + +A `Task` encapsulates the task work of a job. The client interface contains a single synchronous call method that takes an optional memento argument and returns a byte array, which will be packaged with the `CompletedActivity` object surfaced to the `Driver`. An exception may be thrown at any point during the call method; returning control back to the `Evaluator`, which packages the exception and sends it to the `Driver` where it is surfaced as a `FailedActivity`. The `Evaluator` periodically performs a heartbeat with the REEF-AM to convey its status information. A `Task` can optionally implement a method interface that, when called, returns a (bounded) byte array, which the `Evaluator` includes in its heartbeat to the REEF-AM and surfaced to the `Driver`. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/issue-tracker.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/issue-tracker.md b/website/src/site/markdown/issue-tracker.md new file mode 100644 index 0000000..e5ce3ca --- /dev/null +++ b/website/src/site/markdown/issue-tracker.md @@ -0,0 +1,21 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Issue Tracker + +Please visit the [ASF JIRA](https://issues.apache.org/jira/browse/REEF) website for the most updated issues. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/license.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/license.md b/website/src/site/markdown/license.md new file mode 100644 index 0000000..7ebb4c4 --- /dev/null +++ b/website/src/site/markdown/license.md @@ -0,0 +1,221 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Project License + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/mailing-list.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/mailing-list.md b/website/src/site/markdown/mailing-list.md new file mode 100644 index 0000000..da070ca --- /dev/null +++ b/website/src/site/markdown/mailing-list.md @@ -0,0 +1,40 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Mailing List + +- [REEF Dev](#reef-dev) +- [Commits](#commits) + +**Note:** These email addresses are automated. Just writing "Subscribe" or "Unsusbscribe" in both the subject line and the body will suffice and a confirmation email with further instructions will be sent as a reply to your email after some delay. + +###<a name="reef-dev"></a>REEF Dev Mailing List + +This gets traffic from both the JIRAs filed on REEF as well as the general development discussions + +- [Subscribe](mailto:[email protected]) ([email protected]) +- [Unsusbscribe](mailto:[email protected]) ([email protected]) +- [Archive](http://mail-archives.apache.org/mod_mbox/incubator-reef-dev/) + +###<a name="commits"></a>Commits Mailing List + +This email list contains gets all commits made to REEF + +- [Subscribe](mailto:[email protected]) ([email protected]) +- [Unsusbscribe](mailto:[email protected]) ([email protected]) +- [Archive](http://mail-archives.apache.org/mod_mbox/incubator-reef-commits/) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/fa77cc63/website/src/site/markdown/powered-by.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/powered-by.md b/website/src/site/markdown/powered-by.md new file mode 100644 index 0000000..d8d8bdf --- /dev/null +++ b/website/src/site/markdown/powered-by.md @@ -0,0 +1,21 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +#Powered By + +- [Azure Stream Analytics](http://azure.microsoft.com/en-us/services/stream-analytics/) \ No newline at end of file
