Hi mentors (and others),

Had an offline discussion this week in which the question came up how an ASF 
project should best go about running performance/throughput/scalability tests – 
i.e. tests that cannot be run locally and require a repeatable environment.

Some options:
* interested companies run the tests on their own infra and publish results. 
Pretty lame, especially because typically only that company’s engineers can 
access the env and investigate further.
* interested companies donate cash to sponsor compute resources, committers can 
run and investigate the tests. Ideal from tech perspective, but I have no idea 
how that cash would make its way from the ASF to a particular project.
* maybe a middle-ground: interested party that happens to have a public cloud 
offering gives credentials to committers

I am mainly interested to learn if there are other ASF projects (e.g. in the 
Big Data/Hadoop ecosystem) that do something similar. Or if there is an 
ASF-recommended way to do this. Or else, where I could ask this question?


From: Michael Marth <mma...@adobe.com>
Date: Wednesday 3 May 2017 20:57
To: "dev@openwhisk.apache.org" <dev@openwhisk.apache.org>
Subject: Re: Performance tests for OpenWhisk


Quick update: sent the below to users@infra. So far no reaction. The archive is 
here [1] but Bertrand tells me only ASF member have access  - for whatever 



On Fri, Apr 28, 2017 at 2:23 PM, Michael Marth 
<mma...@adobe.com<mailto:mma...@adobe.com>> wrote:
Dear Infra team,

I am enquiring on behalf of the OpenWhisk project (currently in Incubator)

We would like to periodically run performance tests on a distributed
environment (OpenWhisk typically runs on more than 1 machine). So we are
basically looking for an ability to spin up/tear down a number of (virtual)
machines and exclusively use them for a certain amount of time (so that the
VMs are not shared and the performance test results are comparable over
The order of magnitude would be ~5-10 VMs for 1 hour 3 times a week.

I would like to find out if there is an ASF-supported mechanism to do that.
For example, can Infra provide such infrastructure? Or is there a cloud
provider (like Azure) that might sponsor such efforts with VMs? Or maybe
there is an established way for commercial companies that are interested in
an ASF project to sponsor (fund) such tests?

If none of the above exists, then it would also be helpful for us to get to
know how other projects run such sort of tests.

Thanks a lot!


From: Markus Thömmes <markusthoem...@me.com<mailto:markusthoem...@me.com>>
Reply-To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" 
Date: Wednesday 26 April 2017 12:59
To: "dev@openwhisk.apache.org<mailto:dev@openwhisk.apache.org>" 
Subject: Re: Performance tests for OpenWhisk

Hi Michael,

yeah that sounds pretty much spot on. I'd like to have at least 2 VMs with 4+ 
cores and 8GB memory. One VM would host the management stack while one would be 
dedicated to an Invoker only. That way we could assert single-invoker 
performance the easiest.

Thanks for helping!


Am 26. April 2017 um 11:36 schrieb Michael Marth 

Does what I describe reflect what you are looking for?
If yes, I am happy to ask on infra.

Let me know

On 26/04/17 07:52, "Bertrand Delacretaz" 
<bdelacre...@apache.org<mailto:bdelacre...@apache.org>> wrote:

Hi Michael,

On Tue, Apr 25, 2017 at 6:52 PM, Michael Marth 
<mma...@adobe.com<mailto:mma...@adobe.com>> wrote:
...Maybe our mentors can chime in. Has this been discussed in the ASF board or 

Best would be to ask the ASF infrastructure team via
us...@infra.apache.org<mailto:us...@infra.apache.org> - briefly describe what 
you need to see what's


Reply via email to