@Mikhail I must disagree. I have set Nexmark tests as PostCommit mainly because
they were used to discover regressions
on all the beam model as soon as possible: when the new regression commit comes
into master. The graphs (see https://bea
m.apache.org/documentation/sdks/java/nexmark/) help a lot in that way. As
Reuven said, performance testing is more a
second aim even if it is part of regression.
Etienne
Le mardi 14 août 2018 à 15:40 -0700, Mikhail Gryzykhin a écrit :
> @Reuven Lax Thank you for clarifying. It makes sense to keep them as
> post-commits then.
> @Ahmet Altay I don't see them being flaky atm. So we are good here.
>
> I would prefer to separate perf tests mostly because of requirements, such as
> runtime, environment, trigger
> restrictions. Right now we have a separate setup with no GHPRB phrase trigger
> for Nexmark tests.
>
> My understanding was that due to the "performance" nature of Nexmark tests,
> that we can not treat them as post-commit
> tests and have to create special setup for them.
>
> --Mikhail
> Have feedback?
>
> On Mon, Aug 13, 2018 at 10:37 AM Ahmet Altay <[email protected]> wrote:
> > I replied before seeing Reuven's comment. In that case it makes sense to
> > keep them as post commit tests.
> > Are these test flaky because they fail some performance metrics? If that is
> > the case it makes sense to separate them
> > from functional tests.
> > On Mon, Aug 13, 2018 at 10:30 AM, Reuven Lax <[email protected]> wrote:
> > > Nexmark was primarily written as a set of functionality tests, not
> > > performance tests. In fact some of the Nexmark
> > > queries are written in knowingly inefficient ways, in order to better
> > > exercise more features of Beam. We also use
> > > Nexmark as performance tests (mostly out of convenience, because we
> > > already have those tests), however I believe
> > > that is a secondary use.
> > >
> > > Reuven
> > >
> > > On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <[email protected]>
> > > wrote:
> > > > Hi everyone,
> > > >
> > > > As I can understand, a lot of tests in Nexmark set are performance
> > > > tests. I suggest to rename(or split) the set
> > > > to performance tests.
> > > >
> > > > Performance tests are much less reliable compared to post-commit tests
> > > > and should have different requirements.
> > > > Additionally, they are much more flaky.
> > > >
> > > > Splitting out performance tests to separate set will allow us to treat
> > > > failures with lower priority and add more
> > > > tolerance for flakes compared to what we have decided for post-commit
> > > > tests.
> > > >
> > > > This will also be more organic to use different builder from
> > > > PostcommitJobBuilder, since we will want different
> > > > requirements for perf tests.
> > > >
> > > > I do not believe we have a problem with this in current state, but I
> > > > expect this to become an issue in the
> > > > future as amount of perf tests grows.
> > > >
> > > > Regards,
> > > > --Mikhail
> > > >
> > > > Have feedback?
