+1 to rename. I agree with Andrew, flakiness and reliability problems with performance are important problems that needs to be fixed. Do we have a sense of what is making them less reliable?
On Mon, Aug 13, 2018 at 10:19 AM, Jean-Baptiste Onofré <[email protected]> wrote: > +1 > > I think it makes sense to rename. > > Regards > JB > Le 13 août 2018, à 19:14, Pablo Estrada <[email protected]> a écrit: >> >> I think it makes sense to rename. >> >> Also, although we should hold perf tests to a high reliability standard, >> we should also prioritize fixing and triaging PostCommit tests earlier. >> >> Best >> -P. >> >> On Mon, Aug 13, 2018 at 10:10 AM Andrew Pilloud < [email protected]> >> wrote: >> >>> I have no objections with renaming these to Perf instead of PostCommit. >>> >>> I do disagree with your assessment that "Performance tests are much less >>> reliable ... they are much more flaky." I think we should be holding perf >>> tests to the same reliability standards as PostCommit tests. I'm wondering >>> why you think otherwise? >>> >>> Andrew >>> >>> On Mon, Aug 13, 2018, 9:36 AM Mikhail Gryzykhin < [email protected]> >>> wrote: >>> >>>> Hi everyone, >>>> >>>> As I can understand, a lot of tests in Nexmark set are performance >>>> tests. I suggest to rename(or split) the set to performance tests. >>>> >>>> Performance tests are much less reliable compared to post-commit tests >>>> and should have different requirements. Additionally, they are much more >>>> flaky. >>>> >>>> Splitting out performance tests to separate set will allow us to treat >>>> failures with lower priority and add more tolerance for flakes compared to >>>> what we have decided for post-commit tests >>>> <https://beam.apache.org/contribute/postcommits-policies/>. >>>> >>>> This will also be more organic to use different builder from >>>> PostcommitJobBuilder >>>> <https://github.com/apache/beam/tree/master/.test-infra/jenkins>, >>>> since we will want different requirements for perf tests. >>>> >>>> I do not believe we have a problem with this in current state, but I >>>> expect this to become an issue in the future as amount of perf tests grows. >>>> >>>> Regards, >>>> --Mikhail >>>> >>>> Have feedback <http://go/migryz-feedback>? >>>> >>> -- >> Got feedback? go/pabloem-feedback >> <https://goto.google.com/pabloem-feedback> >> >
