Repository: reef Updated Branches: refs/heads/master 3483e3d25 -> 35dc55eb6
[REEF-1505] Add instructions on running IMRU examples This change adds instructions on how to run our IMRU examples. Following the decision we've made for such instructions, we should place it next to the code, so that the instructions can be kept in sync with the code they describe. JIRA: [REEF-1505](https://issues.apache.org/jira/browse/REEF-1505) Pull request: This closes #1384 Project: http://git-wip-us.apache.org/repos/asf/reef/repo Commit: http://git-wip-us.apache.org/repos/asf/reef/commit/35dc55eb Tree: http://git-wip-us.apache.org/repos/asf/reef/tree/35dc55eb Diff: http://git-wip-us.apache.org/repos/asf/reef/diff/35dc55eb Branch: refs/heads/master Commit: 35dc55eb65a2817debc77d018ef5618107c0ab5e Parents: 3483e3d Author: Mariia Mykhailova <[email protected]> Authored: Thu Oct 5 16:55:16 2017 -0700 Committer: jwang98052 <[email protected]> Committed: Sat Oct 14 21:22:45 2017 -0700 ---------------------------------------------------------------------- lang/cs/Org.Apache.REEF.IMRU.Examples/README.md | 46 ++++++++++++++++++++ 1 file changed, 46 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/reef/blob/35dc55eb/lang/cs/Org.Apache.REEF.IMRU.Examples/README.md ---------------------------------------------------------------------- diff --git a/lang/cs/Org.Apache.REEF.IMRU.Examples/README.md b/lang/cs/Org.Apache.REEF.IMRU.Examples/README.md new file mode 100644 index 0000000..b7d2b58 --- /dev/null +++ b/lang/cs/Org.Apache.REEF.IMRU.Examples/README.md @@ -0,0 +1,46 @@ +# Running IMRU Examples + +The easiest way to run IMRU examples is from command line. +Build C# code using your preferred method. + +## Running on local runtime + +Navigate to `<reef>\bin\x64\$(Configuration)\Org.Apache.REEF.IMRU.Examples` (`$(Configuration)` is `Debug` or `Release` based on configuration for which you're building). + +Run `.\Org.Apache.REEF.IMRU.Examples.exe` with first parameter set to `false`. + +## Running on Apache Hadoop YARN + +Log in to a machine in YARN cluster from which you'll run the job. + +Copy the contents of `<reef>\bin\x64\$(Configuration)\Org.Apache.REEF.IMRU.Examples` to this machine. + +Run `.\Org.Apache.REEF.IMRU.Examples.exe` with first parameter set to `true`. + +## Command line parameters + +Command line parameters are parsed based on their position in the list of parameters, so if you need to pass a non-default value for a parameter, you have to pass explicit values for all preceding parameters. + +1. run on YARN (default `false`): if `true`, run the test on YARN, otherwise run the test on local runtime. +2. number of nodes (default `2`): number of nodes on which IMRU job will run. This number includes the master node, i.e. the number of mappers will be one less than the value you pass. +3. start port (default `8900`): the start of port range. +4. port range (default `1000`): the number of ports in port range. Many clusters have firewall configurations which only allow processes to open ports in specific ranges. These are specified here as a start and the length of the port range. REEF will pick a random available port from that range for its communications. +5. test name (default `MapperCount`): the name of the test to run. Can take one of the three values (case-insensitive): +* `MapperCount` (simple job which counts the number of mappers it started), +* `BroadcastAndReduce` (performs broadcast and reduce on a dataset) or +* `BroadcastAndReduceFT` (same with fault tolerance). + +`MapperCount` job has one extra parameter: the name of the remote file to write output to (default value is empty, so the result is written only to local file). + +`BroadcastAndReduce` job performs a basic IMRU job. `Update` sends the current iteration count as a n-dimensional array of integers. `Map` checks that the its input is correct and merely returns it. `Reduce` sums up the arrays. Both plain and fault-tolerant version of `BroadcastAndReduce` accept the following parameters: + +6. `dimensions` (default: `10`): The number of dimensions in the `int[]` array being bounced back and forth between Mapper and Updater. +7. `chunkSize` (default: `2`): The number of dimensions to be processed at once in the Reducers. Note: IMRU performs pipelining. This parameter allows you to play with the effects of that. +8. `mapperMemory` (default: `512`): Amount of memory to be allocated for each map task, in MB. +9. `updateTaskMemory` (default: `512`): Amount of memory to be allocated to the update task, in MB. +10. `iterations` (default: `100`): The number of iterations this example should run for. Set to `1` to measure the overhead of starting the job. + +`BroadcastAndReduceFT` job runs the same workload as `BroadcastAndReduce`, but with some of the nodes forced to fail. This job is used to test fault-tolerance of IMRU scenario. It accepts two extra parameters: + +11. `maxRetryNumberInRecovery` (default 10): The number of times IMRU job will try to recover from failures of some of the mappers. +12. `totalNumberOfForcedFailures` (default 2): The number of retries in which forced mapper failures will be introduced. For jobs running on local runtime, this is the main source of failures. For jobs running on YARN, environment will typically provide sufficient number of natural failures to not need artificial ones.
