This is an automated email from the ASF dual-hosted git repository.
jbarrett pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/geode-benchmarks.git
The following commit(s) were added to refs/heads/develop by this push:
new 439b857 Expanding and clarifying readme files for using aws and
scripts. (#102)
439b857 is described below
commit 439b85762ed5d8261930c82bfc449cd9ab1ef41d
Author: Donal Evans <[email protected]>
AuthorDate: Mon Aug 26 12:11:46 2019 -0700
Expanding and clarifying readme files for using aws and scripts. (#102)
---
README.md | 22 +++++++-------
infrastructure/scripts/aws/README.md | 57 +++++++++++++++++++++++++++++++-----
2 files changed, 59 insertions(+), 20 deletions(-)
diff --git a/README.md b/README.md
index e3f6534..3fad270 100644
--- a/README.md
+++ b/README.md
@@ -13,19 +13,17 @@ test configuration and test execution framework.
These benchmarks are under development.
## Running the benchmarks
-The benchmarks require machines with passwordless ssh enabled in order to run.
-So ensure that the authentication key pair for SSH does not have a passphrase.
If you had
-already previously created a key pair with a passphrase, you can create a
different key pair which
-is of a different type than the previously created one. Public key needs to be
in PEM format. Some newer OpenSSH
-versions default to a new format. Use `-m PEM` to force PEM format.
+The benchmarks require machines with passwordless ssh enabled in order to run,
so ensure that the authentication key pair for SSH does not have a passphrase.
If you have previously created a key pair with a passphrase, you can create a
different key pair which
+is of a different type than the previously created one. Be sure to backup your
existing key pair before creating a new one. The public key needs to be in PEM
format, but some newer OpenSSH
+versions default to a new format. PEM format can be forced by using `-m PEM`:
```
ssh-keygen -m PEM -t rsa
```
-While runinng a test on a single machine (i.e. localhost) add the generated
key to `authorized_keys` to authorize the user.
+While runinng a test on a single machine (i.e. localhost) add the generated
key to `authorized_keys` to authorize the user:
```
cat <your_public_key_file> >> ~/.ssh/authorized_keys
```
-Test if you can ssh to localhost.
+Test if you can ssh to localhost:
```
ssh localhost
```
@@ -59,9 +57,9 @@ Options:
-d : Debug
-i : Info
```
-### Running in aws
+### Scripts for running in aws and analyzing results
-This project includes some scripts to automate running benchmarks in AWS. See
the
+This project includes some scripts to automate running benchmarks in AWS and
analyzing the results produced (as well as the results produced from running
locally). See the
[README.md](infrastructure/scripts/aws/README.md) in the infrastructure/aws
directory.
## Project structure
@@ -72,12 +70,12 @@ The project is divided into two modules
## Sample benchmark
-Benchmarks are defined in a declarative configuration classes. Each
configuration class is run as
+Benchmarks are defined in declarative configuration classes. Each
configuration class is run as
a junit test which calls the configure method and passes it to the TestRunner,
which executes
the test on the provided TEST_HOSTS.
-Benchmarks are composed of `before tasks`, `after tasks`, and `workload
tasks`. The `before` and `after`
-tasks are run once. `Workload` tasks are run repeatedly and their execution
time is measured and
+Benchmarks are composed of `before` tasks, `after` tasks, and `workload`
tasks. Each seperate `before` and `after`
+task is run once. `Workload` tasks are run repeatedly and their execution time
is measured and
reported by the yardstick framework.
```java
diff --git a/infrastructure/scripts/aws/README.md
b/infrastructure/scripts/aws/README.md
index badff0a..177de15 100644
--- a/infrastructure/scripts/aws/README.md
+++ b/infrastructure/scripts/aws/README.md
@@ -4,11 +4,23 @@ These utilities create instances and run tests in your AWS
account
# Prerequisites
* You must have the aws cli installed. If `aws` is not on your path then you
can try to install it with `pip3 install awscli --upgrade --user`. See
[Amazon's aws cli installation
instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
-* You must also set your secret key for the CLI. You must set up a profile
named `geode-benchmarks`, so use the command `aws configure --profile
geode-benchmarks` to configure the CLI. You will need to specify the "AWS
Access Key ID" and "AWS Secret Access Key". You can get these from an existing
team member. Set the "Default region name" to "us-west-2". See [Amazon's
instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+* You must also set your secret key for the CLI. You must set up a profile
named `geode-benchmarks`, so use the command `aws configure --profile
geode-benchmarks` to configure the CLI. You will need to specify the "AWS
Access Key ID" and "AWS Secret Access Key," which can be obtained from a team
member. Set the "Default region name" to "us-west-2". See [Amazon's
instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
+
+ Alternately, if you already have AWS credentials and just need to
create the `geode-benchmarks` profile, you can first modify your existing AWS
credentials file (found at `~/.aws/credentials`) and add the following lines:
+
+ [geode-benchmarks]
+ aws_access_key_id = Your access ID key
+ aws_secret_access_key = Your secret access key
+
+ Then modify the `config` file found in the same directory and add the
following:
+
+ [profile geode-benchmarks]
+ region = us-west-2
+
# Image
-If you need to build the image, you must have packer installed. But you can
run the following scripts (launch, run, destroy) without building the image.
+If you need to build the image, you must have packer installed. The following
scripts (launch, run, analyze, destroy) can be run without building the image.
Build the image in the image directory using the `build_image.sh` script.
@@ -20,13 +32,13 @@ One of the ways that can be done is via environment
variables.
prompt> aws configure
AWS Access Key ID [****************foo1]:
AWS Secret Access Key [****************bar2]:
- Default region name [us-west-1]:
+ Default region name [us-west-2]:
Export environment variables as follows.
export AWS_ACCESS_KEY_ID=myaccesskeyfoo1
export AWS_SECRET_ACCESS_KEY=mysecretaccesskeybar2
- export AWS_REGION="us-west-1"
+ export AWS_REGION="us-west-2"
# launch_cluster.sh
@@ -85,12 +97,29 @@ Options:
-B|--gbb|--baseline-branch|--baseline-geode-branch : Geode Baseline
Branch (default: develop)
-m|--metadata : Test metadata to
output to file, comma-delimited (optional)
--ci : Set when the
instances are being started for use in Continuous Integration
- -- : All subsequent
arguments are passed to the benchmark tast as arguments
+ -- : All subsequent
arguments are passed to the benchmark task as arguments
-h|-?|--help : Help message
e.g. ./run_against_baseline.sh -t test_environment -v <sha1 of target
version> -V <sha1 of base version> -R <baseline repo e.g. user/geode> -B
<baseline branch name> -b <target branch name> -r <target repo e.g. user/geode>
+# analyze_tests.sh
+Compares the results of two benchmark runs and outputs analysis of their
relative performance.
+
+Usage:
+
+ ./analyze_tests.sh [-o <output directory> | [--baselineDir <baseline
directory> --branchDir <branch directory>]] [options ...] [-- arguments ...]
+
+Options:
+
+ -o|--output|--outputDir : The directory containing benchmark results
+ --baseline|--baselineDir : The directory containing baseline benchmark
results
+ --branch|--branchDir : The directory containing branch benchmark
results
+ --ci : Set if starting instances for Continuous
Integration
+ -- : All subsequent arguments are passed to the
benchmark task as arguments.
+ -h|--help : This help message
+
+
# destroy_cluster.sh
Destroys a cluster that you created.
@@ -105,12 +134,24 @@ Options:
-h|-?|--help : Help message
-#Example
+## Example
+
+Example 1 - Generating and comparing two benchmark runs using run_tests.sh and
analyze_tests.sh.
+
+The first command creates a new cluster with 4 instances and the tag
"mycluster" using launch_cluster.sh.
+
+The second command runs only the `MyCustomBenchmark` benchmark test (by using
the `-- --tests=MyCustomBenchmark` argument) found on the `myBenchmarkBranch`
branch of the `myGit/geode-benchmarks` repository. This benchmark runs against
the develop branch of Geode, adds some metadata, and outputs the results to
`~/benchmarking/baseline` using run_test.sh.
+
+The third command runs the same benchmark against the `myGeodeBranch` branch
of the `myGit/geode` repository and outputs the results to
`~/benchmarking/branch` using run_tests.sh.
+
+The fourth command compares the results found in `~/benchmarking/branch` to
the results found in `~/benchmarking/baseline` and outputs analysis of the
operations per second and latency of the branch benchmark relative to the
baseline using analyze_tests.sh.
-Example 1 - run_test.sh:
+The fifth command destroys the cluster using destroy_cluster.sh.
```bash
./launch_cluster.sh --tag mycluster --count 4
-./run_tests.sh --tag mycluster --geode-branch develop --benchmark-branch
benchmarkBranch --metadata
"'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'"
+./run_tests.sh --tag mycluster --geode-branch develop --benchmark-repo
myGit/geode-benchmarks --benchmark-branch myBenchmarkBranch --metadata
"'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output
~/benchmarking/baseline -- --tests=MyCustomBenchmark
+./run_tests.sh --tag mycluster --geode-repo myGit/geode --geode-branch
myGeodeBranch --benchmark-repo myGit/geode-benchmarks --benchmark-branch
myBenchmarkBranch --metadata
"'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output
~/benchmarking/branch -- --tests=MyCustomBenchmark
+./analyze_tests.sh --branch ~/benchmarking/branch --baseline
~/benchmarking/baseline
./destroy_cluster.sh --tag mycluster
```
