[
https://issues.apache.org/jira/browse/BEAM-6097?focusedWorklogId=175084&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175084
]
ASF GitHub Bot logged work on BEAM-6097:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Dec/18 22:46
Start Date: 13/Dec/18 22:46
Worklog Time Spent: 10m
Work Description: melap commented on a change in pull request #7236:
[BEAM-6097] Nemo Runner
URL: https://github.com/apache/beam/pull/7236#discussion_r241587867
##########
File path: website/src/documentation/runners/nemo.md
##########
@@ -0,0 +1,169 @@
+---
+layout: section
+title: "Apache Nemo Runner"
+section_menu: section-menu/runners.html
+permalink: /documentation/runners/nemo/
+redirect_from: /learn/runners/nemo/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Using the Apache Nemo Runner
+
+The Apache Nemo Runner can be used to execute Beam pipelines using [Apache
Nemo](http://nemo.apache.org).
+The Nemo Runner can optimize Beam pipelines with the Nemo compiler through
various optimization passes
+and execute them in distributed fashion using the Nemo runtime. Deploying a
self-contained application
+for local mode or running using resource managers like YARN or Mesos are
possible.
+
+The Nemo Runner executes Beam pipelines on top of Apache Nemo, providing:
+
+* Batch and streaming pipelines
+* Fault-tolerance
+* Integration with YARN and other components of the Apache Hadoop ecosystem
+* Support for the various optimizations provided by the Nemo optimizer
+
+The [Beam Capability Matrix]({{ site.baseurl
}}/documentation/runners/capability-matrix/) documents the
+supported capabilities of the Nemo Runner.
+
+## Nemo Runner prerequisites and setup
+
+The Nemo Runner can be used simply by adding a dependency on a version of the
Nemo runner newer than `0.1`
+to your pom.xml as followings:
+
+```
+<dependency>
+ <groupId>org.apache.nemo</groupId>
+ <artifactId>nemo-compiler-frontend-beam</artifactId>
+ <version>${nemo.version}</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+</dependency>
+```
+
+## Deploying Nemo with your Application
+
+A self-contained application could be easier to manage and fully use the
functionalities that Nemo provides.
+Simply add the dependency shown above and shade the application jar using the
maven shade plugin:
+
+```
+<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadedArtifactAttached>true</shadedArtifactAttached>
+ <shadedClassifierName>shaded</shadedClassifierName>
+ <transformers>
+ <transformer
+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+</plugin>
+```
+
+After running `mvn package`, run `ls target` and you should see (when your
artifactId is `beam-examples`
+and the version is `1.0.0`):
+
+```
+beam-examples-1.0.0-shaded.jar
+```
+
+With this shaded jar, you may use the `bin/run_beam.sh` shell script as
following:
+
+```
+## MapReduce example
+./bin/run_beam.sh \
+ -job_id mr_default \
+ -user_main org.apache.nemo.examples.beam.WordCount \
+ -user_args "`pwd`/examples/resources/test_input_wordcount
`pwd`/examples/resources/test_output_wordcount"
+```
+
+To use Nemo using YARN, set the `-deploy_mode` flag on Nemo to `yarn`.
+
+More instructions can be seen on the README of the [Apache Nemo
GitHub](https://github.com/apache/incubator-nemo).
+
+## Pipeline Options for the Nemo Runner
+
+When executing your pipeline with the Nemo Runner, you should consider the
following pipeline options
+
+| Field | Description | Default Value |
+| ------------- |---------------| -----:|
+| `runner` | The pipeline runner to use. This option allows you to
determine the pipeline runner at runtime. | Set to `NemoRunner` to run using
Nemo |
Review comment:
```suggestion
| `runner` | The pipeline runner to use. This option allows you to
determine the pipeline runner at runtime. | Set to `NemoRunner` to run using
Nemo. |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 175084)
> Add NemoRunner
> --------------
>
> Key: BEAM-6097
> URL: https://issues.apache.org/jira/browse/BEAM-6097
> Project: Beam
> Issue Type: New Feature
> Components: runner-ideas
> Reporter: Won Wook SONG
> Assignee: Won Wook SONG
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Add NemoRunner (http://nemo.apache.org)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
