markusthoemmes closed pull request #3475: Move performance repository to core repository. URL: https://github.com/apache/incubator-openwhisk/pull/3475
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/performance/.travis.yml b/performance/.travis.yml new file mode 100644 index 0000000000..b425c604d3 --- /dev/null +++ b/performance/.travis.yml @@ -0,0 +1,14 @@ +sudo: required +language: java +services: + - docker + +env: + - TERM=dumb + +install: + - ./preparation/deploy.sh + +script: + - ./wrk_tests/latency.sh "http://172.17.0.1:10001"; "$(cat openwhisk/ansible/files/auth.guest)" 2m + - ./wrk_tests/throughput.sh "http://172.17.0.1:10001"; "$(cat openwhisk/ansible/files/auth.guest)" 4 2 2m diff --git a/performance/README.md b/performance/README.md new file mode 100644 index 0000000000..c1d668f394 --- /dev/null +++ b/performance/README.md @@ -0,0 +1,33 @@ +# :electric_plug: Apache OpenWhisk - Performance Tests +A few simple but efficient test suites for determining the maximum throughput and end-user latency of the Apache OpenWhisk system. + +## Workflow +- A standard OpenWhisk system is deployed. (_Note that the edge NGINX router and API Gateway are currently left out. As a consequence, the tests talk directly to the controller._) +- All limits are set to 999999, which in our current use case means "No throttling at all". +- The deployment is using the docker setup proposed by the OpenWhisk development team: `overlay` driver and HTTP API enabled via a UNIX port. + +The load is driven by the blazingly fast [`wrk`](https://github.com/wg/wrk). + +#### Travis Machine Setup +The [machine provided by Travis](https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments) has ~2 CPU cores (likely shared through virtualization) and 7.5GB memory. + +## Suites + +### Latency Test +Determines the end-to-end latency a user experience when doing a blocking invocation. The action used is a no-op so the numbers returned are the plain overhead of the OpenWhisk system. + +- 1 HTTP request at a time (concurrency: 1) +- You can specify how long this test will run. Default are 30s. +- no-op action + +**Note:** The throughput number has a 100% correlation with the latency in this case. This test does not serve to determine the maximum throughput of the system. + +### Throughput Test +Determines the maximum throughput a user can get out of the system while using a single action. The action used is a no-op, so the numbers are plain OpenWhisk overhead. Note that the throughput does not directly correlate to end-to-end latency here, as the system does more processing in the background as it shows to the user in a blocking invocation. + +- 4 HTTP requests at a time (concurrency: 4) (using CPU cores * 2 to exploit some buffering) +- 10.000 samples with a single user +- no-op action + +## Running tests against your own system is simple too! +All you have to do is use the corresponding script located in /*_tests folder, remembering that the parameters are defined inline. diff --git a/performance/preparation/create.sh b/performance/preparation/create.sh new file mode 100755 index 0000000000..1a796499ec --- /dev/null +++ b/performance/preparation/create.sh @@ -0,0 +1,17 @@ +#!/bin/sh +set -e + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# Name of the action to create and test. +action=$3 + +# create a noop action +echo "Creating action $action" +curl -k -u "$credentials" "$host/api/v1/namespaces/_/actions/$action" -XPUT -d '{"namespace":"_","name":"test","exec":{"kind":"nodejs:default","code":"function main(){return {};}"}}' -H "Content-Type: application/json" + +# run the noop action +echo "Running $action once to assert an intact system" +curl -k -u "$credentials" "$host/api/v1/namespaces/_/actions/$action?blocking=true" -XPOST diff --git a/performance/preparation/deploy.sh b/performance/preparation/deploy.sh new file mode 100755 index 0000000000..56281768b8 --- /dev/null +++ b/performance/preparation/deploy.sh @@ -0,0 +1,25 @@ +#!/bin/sh + +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# common docker setup +sudo gpasswd -a travis docker +sudo -E bash -c 'echo '\''DOCKER_OPTS="-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock --storage-driver=overlay --userns-remap=default"'\'' > /etc/default/docker' +sudo service docker restart + +# install ansible +pip install --user ansible==2.4.2.0 + +cd $currentDir/../../ansible +ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=openwhisk -e docker_registry=docker.io/ -e limit_invocations_per_minute=999999 -e limit_invocations_concurrent=999999 -e limit_invocations_concurrent_system=999999" + +$ANSIBLE_CMD setup.yml +$ANSIBLE_CMD prereq.yml +$ANSIBLE_CMD couchdb.yml +$ANSIBLE_CMD initdb.yml +$ANSIBLE_CMD wipe.yml + +$ANSIBLE_CMD kafka.yml +$ANSIBLE_CMD controller.yml +$ANSIBLE_CMD invoker.yml diff --git a/performance/wrk_tests/latency.sh b/performance/wrk_tests/latency.sh new file mode 100755 index 0000000000..bee706b1f6 --- /dev/null +++ b/performance/wrk_tests/latency.sh @@ -0,0 +1,12 @@ +#!/bin/sh +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# How long to run the test +duration=${3:-30s} + +$currentDir/throughput.sh $host $credentials 1 1 $duration diff --git a/performance/wrk_tests/post.lua b/performance/wrk_tests/post.lua new file mode 100644 index 0000000000..e3e664cb8c --- /dev/null +++ b/performance/wrk_tests/post.lua @@ -0,0 +1 @@ +wrk.method = "POST" \ No newline at end of file diff --git a/performance/wrk_tests/throughput.sh b/performance/wrk_tests/throughput.sh new file mode 100755 index 0000000000..6ed245332e --- /dev/null +++ b/performance/wrk_tests/throughput.sh @@ -0,0 +1,31 @@ +#!/bin/sh +set -e +currentDir="$(cd "$(dirname "$0")"; pwd)" + +# Host to use. Needs to include the protocol. +host=$1 +# Credentials to use for the test. USER:PASS format. +credentials=$2 +# concurrency level of the throughput test: How many requests should +# open in parallel. +concurrency=$3 +# How many threads to utilize, directly correlates to the number +# of CPU cores +threads=${4:-4} +# How long to run the test +duration=${5:-30s} + +action="noopThroughput" +"$currentDir/../preparation/create.sh" "$host" "$credentials" "$action" + +# run throughput tests +encodedAuth=$(echo "$credentials" | tr -d '\n' | base64 | tr -d '\n') +docker run --pid=host --userns=host --rm -v "$currentDir":/data williamyeh/wrk \ + --threads "$threads" \ + --connections "$concurrency" \ + --duration "$duration" \ + --header "Authorization: basic $encodedAuth" \ + "$host/api/v1/namespaces/_/actions/$action?blocking=true" \ + --latency \ + --timeout 10s \ + --script post.lua ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
