I have no objections with renaming these to Perf instead of PostCommit. I do disagree with your assessment that "Performance tests are much less reliable ... they are much more flaky." I think we should be holding perf tests to the same reliability standards as PostCommit tests. I'm wondering why you think otherwise?
Andrew On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin <[email protected]> wrote: > Hi everyone, > > As I can understand, a lot of tests in Nexmark set are performance tests. > I suggest to rename(or split) the set to performance tests. > > Performance tests are much less reliable compared to post-commit tests and > should have different requirements. Additionally, they are much more flaky. > > Splitting out performance tests to separate set will allow us to treat > failures with lower priority and add more tolerance for flakes compared to > what we have decided for post-commit tests > <https://beam.apache.org/contribute/postcommits-policies/>. > > This will also be more organic to use different builder from > PostcommitJobBuilder > <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, since > we will want different requirements for perf tests. > > I do not believe we have a problem with this in current state, but I > expect this to become an issue in the future as amount of perf tests grows. > > Regards, > --Mikhail > > Have feedback <http://go/migryz-feedback>? >
