This is an automated email from the ASF dual-hosted git repository. markusthoemmes pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git
The following commit(s) were added to refs/heads/master by this push: new f3f1cb9 Move performance repository to core repository. (#3475) f3f1cb9 is described below commit f3f1cb9171e0b95a27ac8f43dd8eafd5bce2f938 Author: Christian Bickel <git...@cbickel.de> AuthorDate: Wed Apr 4 15:55:05 2018 +0200 Move performance repository to core repository. (#3475) Co-authored-by: Markus Thömmes <markusthoem...@me.com> Co-authored-by: Jeremias Werner <jeremias.wer...@gmail.com> Co-authored-by: Alexandre Nicastro <kryn...@gmail.com> --- performance/.travis.yml | 14 ++++++++++++++ performance/README.md | 33 +++++++++++++++++++++++++++++++++ performance/preparation/create.sh | 17 +++++++++++++++++ performance/preparation/deploy.sh | 25 +++++++++++++++++++++++++ performance/wrk_tests/latency.sh | 12 ++++++++++++ performance/wrk_tests/post.lua | 1 + performance/wrk_tests/throughput.sh | 31 +++++++++++++++++++++++++++++++ 7 files changed, 133 insertions(+) diff --git a/performance/.travis.yml b/performance/.travis.yml new file mode 100644 index 0000000..b425c60 --- /dev/null +++ b/performance/.travis.yml @@ -0,0 +1,14 @@ +sudo: required +language: java +services: + - docker + +env: + - TERM=dumb + +install: + - ./preparation/deploy.sh + +script: + - ./wrk_tests/latency.sh "http://172.17.0.1:10001" "$(cat openwhisk/ansible/files/auth.guest)" 2m + - ./wrk_tests/throughput.sh "http://172.17.0.1:10001" "$(cat openwhisk/ansible/files/auth.guest)" 4 2 2m diff --git a/performance/README.md b/performance/README.md new file mode 100644 index 0000000..c1d668f --- /dev/null +++ b/performance/README.md @@ -0,0 +1,33 @@ +# :electric_plug: Apache OpenWhisk - Performance Tests +A few simple but efficient test suites for determining the maximum throughput and end-user latency of the Apache OpenWhisk system. + +## Workflow +- A standard OpenWhisk system is deployed. (_Note that the edge NGINX router and API Gateway are currently left out. As a consequence, the tests talk directly to the controller._) +- All limits are set to 999999, which in our current use case means "No throttling at all". +- The deployment is using the docker setup proposed by the OpenWhisk development team: `overlay` driver and HTTP API enabled via a UNIX port. + +The load is driven by the blazingly fast [`wrk`](https://github.com/wg/wrk). + +#### Travis Machine Setup +The [machine provided by Travis](https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments) has ~2 CPU cores (likely shared through virtualization) and 7.5GB memory. + +## Suites + +### Latency Test +Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system. + +- 1 HTTP request at a time (concurrency: 1) +- You can specify how long this test will run. Default are 30s. +- no-op action + +**Note:** The throughput number has a 100% correlation with the latency in this case. This test does not serve to determine the maximum throughput of the system. + +### Throughput Test +Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation. + +- 4 HTTP requests at a time (concurrency: 4) (using CPU cores * 2 to exploit some buffering) +- 10.000 samples with a single user +- no-op action + +## Running tests against your own system is simple too! +All you have to do is use the corresponding script located in /*_tests folder, remembering that the parameters are defined inline. diff --git a/performance/preparation/create.sh b/performance/preparation/create.sh new file mode 100755 index 0000000..1a79649 --- /dev/null +++ b/performance/preparation/create.sh @@ -0,0 +1,17 @@ +#!/bin/sh +set -e + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# Name of the action to create and test. +action=$3 + +# create a noop action +echo "Creating action $action" +curl -k -u "$credentials" "$host/api/v1/namespaces/_/actions/$action" -XPUT -d '{"namespace":"_","name":"test","exec":{"kind":"nodejs:default","code":"function main(){return {};}"}}' -H "Content-Type: application/json" + +# run the noop action +echo "Running $action once to assert an intact system" +curl -k -u "$credentials" "$host/api/v1/namespaces/_/actions/$action?blocking=true" -XPOST diff --git a/performance/preparation/deploy.sh b/performance/preparation/deploy.sh new file mode 100755 index 0000000..5628176 --- /dev/null +++ b/performance/preparation/deploy.sh @@ -0,0 +1,25 @@ +#!/bin/sh + +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# common docker setup +sudo gpasswd -a travis docker +sudo -E bash -c 'echo '\''DOCKER_OPTS="-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock --storage-driver=overlay --userns-remap=default"'\'' > /etc/default/docker' +sudo service docker restart + +# install ansible +pip install --user ansible==2.4.2.0 + +cd $currentDir/../../ansible +ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=openwhisk -e docker_registry=docker.io/ -e limit_invocations_per_minute=999999 -e limit_invocations_concurrent=999999 -e limit_invocations_concurrent_system=999999" + +$ANSIBLE_CMD setup.yml +$ANSIBLE_CMD prereq.yml +$ANSIBLE_CMD couchdb.yml +$ANSIBLE_CMD initdb.yml +$ANSIBLE_CMD wipe.yml + +$ANSIBLE_CMD kafka.yml +$ANSIBLE_CMD controller.yml +$ANSIBLE_CMD invoker.yml diff --git a/performance/wrk_tests/latency.sh b/performance/wrk_tests/latency.sh new file mode 100755 index 0000000..bee706b --- /dev/null +++ b/performance/wrk_tests/latency.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# How long to run the test +duration=${3:-30s} + +$currentDir/throughput.sh $host $credentials 1 1 $duration diff --git a/performance/wrk_tests/post.lua b/performance/wrk_tests/post.lua new file mode 100644 index 0000000..e3e664c --- /dev/null +++ b/performance/wrk_tests/post.lua @@ -0,0 +1 @@ +wrk.method = "POST" \ No newline at end of file diff --git a/performance/wrk_tests/throughput.sh b/performance/wrk_tests/throughput.sh new file mode 100755 index 0000000..6ed2453 --- /dev/null +++ b/performance/wrk_tests/throughput.sh @@ -0,0 +1,31 @@ +#!/bin/sh +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# concurrency level of the throughput test: How many requests should +# open in parallel. +concurrency=$3 +# How many threads to utilize, directly correlates to the number +# of CPU cores +threads=${4:-4} +# How long to run the test +duration=${5:-30s} + +action="noopThroughput" +"$currentDir/../preparation/create.sh" "$host" "$credentials" "$action" + +# run throughput tests +encodedAuth=$(echo "$credentials" | tr -d '\n' | base64 | tr -d '\n') +docker run --pid=host --userns=host --rm -v "$currentDir":/data williamyeh/wrk \ + --threads "$threads" \ + --connections "$concurrency" \ + --duration "$duration" \ + --header "Authorization: basic $encodedAuth" \ + "$host/api/v1/namespaces/_/actions/$action?blocking=true" \ + --latency \ + --timeout 10s \ + --script post.lua -- To stop receiving notification emails like this one, please contact markusthoem...@apache.org.