Repository: incubator-distributedlog Updated Branches: refs/heads/master 878ae87be -> 50f6bd40a
Add documentation for distributedlog-benchmark Add documentation for distributedlog-benchmark - quick overview of how to run dbench. Author: Leigh Stewart <lstew...@twitter.com> Reviewers: Sijie Guo <si...@apache.org> Closes #44 from leighst/lstewart/docs_benchmark Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/50f6bd40 Tree: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/50f6bd40 Diff: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/50f6bd40 Branch: refs/heads/master Commit: 50f6bd40a629800723a50feb66ff3de27b664ef3 Parents: 878ae87 Author: Leigh Stewart <lstew...@twitter.com> Authored: Wed Nov 16 20:29:19 2016 -0800 Committer: Franck Cuny <fc...@apache.org> Committed: Wed Nov 16 20:29:19 2016 -0800 ---------------------------------------------------------------------- docs/admin_guide/loadtest.rst | 100 +++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/50f6bd40/docs/admin_guide/loadtest.rst ---------------------------------------------------------------------- diff --git a/docs/admin_guide/loadtest.rst b/docs/admin_guide/loadtest.rst new file mode 100644 index 0000000..e7a1d6b --- /dev/null +++ b/docs/admin_guide/loadtest.rst @@ -0,0 +1,100 @@ +--- +layout: default + +# Top navigation +top-nav-group: admin-guide +top-nav-pos: 2 +top-nav-title: Load Test + +# Sub-level navigation +sub-nav-group: admin-guide +sub-nav-parent: admin-guide +sub-nav-id: performance +sub-nav-pos: 2 +sub-nav-title: Load Test +--- + +.. contents:: Load Test + +Load Test +========= + +Overview +-------- + +Under distributedlog-benchmark you will find a set of applications intended for generating large amounts of load in a distributedlog custer. These applications are suitable for load testing, performance testing, benchmarking, or even simply smoke testing a distributedlog cluster. + +The dbench script can run in several modes: + +1. bkwrite - Benchmark the distributedlog write path using the core library + +2. write - Benchmark the distributedlog write path, via write proxy, using the thin client + +3. read - Benchmark the distributedlog read path using the core library + + +Running Dbench +-------------- + +The distributedlog-benchmark binary dbench is intended to be run simultaneously from many machines with identical settings. Together, all instances of dbench comprise a benchmark job. How you launch a benchmark job will depend on your operating environment. We recommend using a cluster scheduler like aurora or kubernetes to simplify the process, but tools like capistrano can also simplify this process greatly. + +The benchmark script can be found at + +:: + + distributedlog-benchmark/bin/dbench + +Arguments may be passed to this script via environment variables. The available arguments depend on the execution mode. For an up to date list, check the script itself. + + +Write to Proxy with Thin Client +------------------------------- + +The proxy write test (mode = `write`) can be used to send writes to a proxy cluster to be written to a set of streams. + +For example to use the proxy write test to generate 10000 requests per second across 10 streams using 50 machines, run the following command on each machine. + +:: + + STREAM_NAME_PREFIX=loadtest_ + BENCHMARK_DURATION=3600 # seconds + DL_NAMESPACE=<dl namespace> + NUM_STREAMS=10 + INITIAL_RATE=200 + distributedlog-benchmark/bin/dbench write + + +Write to BookKeeper with Core Library +------------------------------------- + +The core library write test (mode = `bkwrite`) can be used to send writes to directly to bookkeeper using the core library. + +For example to use the core library write test to generate 100MBps across 10 streams using 100 machines, run the following command on each machine. + +:: + + STREAM_NAME_PREFIX=loadtest_ + BENCHMARK_DURATION=3600 # seconds + DL_NAMESPACE=<dl namespace> + NUM_STREAMS=10 + INITIAL_RATE=1024 + MSG_SIZE=1024 + distributedlog-benchmark/bin/dbench bkwrite + + +Read from BookKeeper with Core Library +-------------------------------------- + +The core library read test (mode = `read`) can be used to read directly from bookkeeper using the core library. + +For example to use the core library read test to read from 10 streams on 100 instances, run the following command on each machine. + +:: + + STREAM_NAME_PREFIX=loadtest_ + BENCHMARK_DURATION=3600 # seconds + DL_NAMESPACE=<dl namespace> + MAX_STREAM_ID=9 + NUM_READERS_PER_STREAM=5 + TRUNCATION_INTERVAL=60 # seconds + distributedlog-benchmark/bin/dbench read
