Hi Markus - I found the issue with slow throughput performance I was seeing in the new-containerpool branch of your fork, I created a PR in a separate fork here, not sure of a good way to do a PR back to your fork (multiple forks etc), but the change is pretty minor - just delay some time before making a container eligible for removal (when scheduling a "free" container fails) https://github.com/tysonnorris/openwhisk-new-containerpool/pull/1
Test results below are based on new-containerpool branch + this change, compared to master. Approximately double throughput. To be clear this is *without* concurrent execution of actions, which I would still like to look into a bit further. Thanks Tyson new-containerpool throughput.sh -c 40 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Max requests: 10000 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Concurrency level: 40 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Agent: keepalive [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Completed requests: 10000 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Total errors: 0 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Total time: 156.21377541599998 s [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Requests per second: 64 [Wed May 03 2017 15:08:49 GMT+0000 (UTC)] INFO Mean latency: 623.7 ms new-containerpool throughput.sh -c 10 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Max requests: 10000 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Concurrency level: 10 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Agent: keepalive [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Completed requests: 10000 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Total errors: 0 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Total time: 158.92872544899998 s [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Requests per second: 63 [Wed May 03 2017 15:11:50 GMT+0000 (UTC)] INFO Mean latency: 158.8 ms master, throughput -c 40 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Max requests: 10000 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Concurrency level: 40 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Agent: keepalive [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Completed requests: 10000 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Total errors: 0 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Total time: 417.37680744 s [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Requests per second: 24 [Wed May 03 2017 15:31:13 GMT+0000 (UTC)] INFO Mean latency: 1666.6 ms master, throughput -c 10 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Max requests: 10000 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Concurrency level: 10 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Agent: keepalive [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Completed requests: 10000 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Total errors: 0 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Total time: 349.43771015699997 s [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Requests per second: 29 [Wed May 03 2017 15:43:12 GMT+0000 (UTC)] INFO Mean latency: 349.2 ms
