Hi All - I created this issue some time ago to discuss concurrent requests on actions: [1] Some people mentioned discussing on the mailing list so I wanted to start that discussion.
I’ve been doing some testing against this branch with Markus’s work on the new container pool: [2] I believe there are a few open PRs in upstream related to this work, but this seemed like a reasonable place to test against a variety of the reactive invoker and pool changes - I’d be interested to hear if anyone disagrees. Recently I ran some tests - with “throughput.sh” in [3] using concurrency of 10 (it will also be interesting to test with the --rps option in loadtest...) - using a change that checks actions for an annotation “max-concurrent” (in case there is some reason actions want to enforce current behavior of strict serial invocation per container?) - when scheduling an actions against the pool, if there is a currently “busy” container with this action, AND the annotation is present for this action, AND concurrent requests < max-concurrent, the this container is used to invoke the action Below is a summary (approx 10x throughput with concurrent requests) and I would like to get some feedback on: - what are the cases for having actions that require container isolation per request? node is a good example that should NOT need this, but maybe there are cases where it is more important, e.g. if there are cases where stateful actions are used? - log collection approach: I have not attempted to resolve log collection issues; I would expect that revising the log sentinel marker to include the activation ID would help, and logs stored with the activation would include interleaved activations in some cases (which should be expected with concurrent request processing?), and require some different logic to process logs after an activation completes (e.g. logs emitted at the start of an activation may have already been collected as part of another activation’s log collection, etc). - advice on creating a PR to discuss this in more detail - should I wait for more of the container pooling changes to get to master? Or submit a PR to Markus’s new-containerpool branch? Thanks Tyson Summary of loadtest report with max-concurrent ENABLED (I used 10000, but this limit wasn’t reached): [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughputConcurrent?blocking=true [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Max requests: 10000 [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Concurrency level: 10 [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Agent: keepalive [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Completed requests: 10000 [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Total errors: 0 [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Total time: 241.900480915 s [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Requests per second: 41 [Sat Apr 29 2017 16:32:37 GMT+0000 (UTC)] INFO Mean latency: 241.7 ms Summary of loadtest report with max-concurrent DISABLED: [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Target URL: https://192.168.99.100/api/v1/namespaces/_/actions/noopThroughput?blocking=true [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Max requests: 10000 [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Concurrency level: 10 [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Agent: keepalive [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Completed requests: 10000 [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Total errors: 19 [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Total time: 2770.658048791 s [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Requests per second: 4 [Sat Apr 29 2017 19:21:51 GMT+0000 (UTC)] INFO Mean latency: 2767.3 ms [1] https://github.com/openwhisk/openwhisk/issues/2026 [2] https://github.com/markusthoemmes/openwhisk/tree/new-containerpool [3] https://github.com/markusthoemmes/openwhisk-performance
