I could pass all tests without disabling any performance tests by opening a new PR[1].
It's ready to review now. I think setting up a new environment for the performance test is orthogonal to my change and would be performed by this issue[2]. [1] https://github.com/apache/openwhisk/pull/5192 [2] https://github.com/apache/openwhisk/issues/5190 2022년 1월 5일 (수) 오전 9:19, Dominic Kim <style9...@gmail.com>님이 작성: > The tests with wrk are comprised of 6 different tests. > The first four utilize fewer concurrencies while the other two utilize 100 > concurrencies. > After changing the pause grace, it is more resilient with lower > concurrency tests but still be failed with higher concurrency. > > One strange thing is I opened a new PR copying the original one, and it > passes the tests. > https://github.com/apache/openwhisk/pull/5189 > > No matter how many I tried the original PR did not pass the tests, but > with the new one with the same change, it passed the tests. > It however does not mean the performance test is working as expected, > there are still failed activations. > It just does not go into an unresponsive state. > > I left a query to the Travis team asking if there are any differences > between the two. > > New PRs might pass the tests and we can disable tests with higher > concurrency only for now. > But I feel eventually it's better to run them in another environment as > tests are actually being failed. > > I confirmed the VM(openwhisk-vm1-he-de.apache.org) we have has 8 cores > with 32GB memory. > But it seems it is being used as a Jenkins slave. > I am not quite sure if we can just use it or should we ask for another VM. > > -dom > > > > > 2022년 1월 5일 (수) 오전 5:53, Davi Ribeiro <daviantune...@gmail.com>님이 작성: > >> hi, >> >> What you think about reduce the load in performance test in wrk to see >> what >> happens? >> >> Davi Ribeiro >> >> >> >> Em seg., 3 de jan. de 2022 às 21:26, Dominic Kim <style9...@gmail.com> >> escreveu: >> >> > Dear whiskers. >> > >> > I've found the performance tests have not been working so far. >> > According to the following test, it is shown it passed the test but >> > actually, there are some activations with other than 2xx,3xx response. >> > https://app.travis-ci.com/github/apache/openwhisk/jobs/479539830 >> > >> > Even with some wrong results, there was no issue running the subsequent >> > Gatling tests so far. >> > But now, the system becomes unavailable after running tests with `wrk`. >> > >> > I got inside the VM and observed status of runtime containers kept >> changing >> > between running and paused. >> > I suspect this is because the VM has too little resources to run all OW >> > components and the wrk client so it could not generate/forward >> activations >> > to the invoker in time. >> > And such Docker workloads made the Docker engine become abnormal. >> > I have no idea what has changed on the Travis VM side yet, but at >> least, I >> > found the docker version is changed. >> > >> > I am not sure it is effective to run performance tests on a small VM >> with 2 >> > cores and 8GB memory. >> > Now it is rather an obstacle to merging new PRs. >> > >> > I propose temporarily disabling the performance tests until we find a >> > better way. >> > Maybe we can utilize a bigger VM provided by ASF. >> > >> > Please share your opinion. >> > >> > Thanks >> > Regards >> > Dominic >> > >> > Reference: >> > https://github.com/apache/openwhisk/pull/5158#issuecomment-1003900935 >> > >> >
