I think it makes sense to rename. Also, although we should hold perf tests to a high reliability standard, we should also prioritize fixing and triaging PostCommit tests earlier.
Best -P. On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud <[email protected]> wrote: > I have no objections with renaming these to Perf instead of PostCommit. > > I do disagree with your assessment that "Performance tests are much less > reliable ... they are much more flaky." I think we should be holding perf > tests to the same reliability standards as PostCommit tests. I'm wondering > why you think otherwise? > > Andrew > > On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin <[email protected]> wrote: > >> Hi everyone, >> >> As I can understand, a lot of tests in Nexmark set are performance tests. >> I suggest to rename(or split) the set to performance tests. >> >> Performance tests are much less reliable compared to post-commit tests >> and should have different requirements. Additionally, they are much more >> flaky. >> >> Splitting out performance tests to separate set will allow us to treat >> failures with lower priority and add more tolerance for flakes compared to >> what we have decided for post-commit tests >> <https://beam.apache.org/contribute/postcommits-policies/>. >> >> This will also be more organic to use different builder from >> PostcommitJobBuilder >> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since >> we will want different requirements for perf tests. >> >> I do not believe we have a problem with this in current state, but I >> expect this to become an issue in the future as amount of perf tests grows. >> >> Regards, >> --Mikhail >> >> Have feedback <http://go/migryz-feedback>? >> > -- Got feedback? go/pabloem-feedback
