Re: Finer-grained test runs?

2020-07-13 Thread Kenneth Knowles
Some context links for the benefit of the thread & archive: Beam issue mentioning a Jenkins plugin that caches on the Jenkins master: https://issues.apache.org/jira/browse/BEAM-4400 Beam's request to infra: https://issues.apache.org/jira/browse/INFRA-16630 Denied and reasoning on prior request:

Re: Finer-grained test runs?

2020-07-13 Thread Kenneth Knowles
Having thought this over a bit, I think there are a few goals and they are interfering with each other. 1. Clear signal for module / test suite health. This is a post-commit concern. Post-commit jobs already all run as cronjobs with no dependency-driven stuff. 2. Making precommit test signal stay

Re: Finer-grained test runs?

2020-07-10 Thread Kenneth Knowles
On Thu, Jul 9, 2020 at 1:44 PM Robert Bradshaw wrote: > I wonder how hard it would be to track greenness and flakiness at the > level of gradle project (or even lower), viewed hierarchically. > Looks like this is part of the Gradle Enterprise Tests Dashboard offering:

Re: Finer-grained test runs?

2020-07-09 Thread Robert Bradshaw
It does sound like we're generally on the same page. Minor comments below. On Thu, Jul 9, 2020 at 1:00 PM Kenneth Knowles wrote: > > On Thu, Jul 9, 2020 at 11:47 AM Robert Bradshaw wrote: >> >> On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: >> > >> >> If Brian's: it does not result in

Re: Finer-grained test runs?

2020-07-09 Thread Luke Cwik
No, not without doing the research myself to see what is the current tooling available. On Thu, Jul 9, 2020 at 1:17 PM Kenneth Knowles wrote: > > > On Thu, Jul 9, 2020 at 1:10 PM Luke Cwik wrote: > >> The budget would represent some criteria that we need from tests (e.g. >> percent passed, max

Re: Finer-grained test runs?

2020-07-09 Thread Kenneth Knowles
On Thu, Jul 9, 2020 at 1:10 PM Luke Cwik wrote: > The budget would represent some criteria that we need from tests (e.g. > percent passed, max num skipped tests, test execution time, ...). If we > fail the criteria then there must be actionable work (such as fix tests) > followed with something

Re: Finer-grained test runs?

2020-07-09 Thread Luke Cwik
The budget would represent some criteria that we need from tests (e.g. percent passed, max num skipped tests, test execution time, ...). If we fail the criteria then there must be actionable work (such as fix tests) followed with something that prevents the status quo from continuing (such as

Re: Finer-grained test runs?

2020-07-09 Thread Kenneth Knowles
On Thu, Jul 9, 2020 at 11:47 AM Robert Bradshaw wrote: > On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > > > >> If Brian's: it does not result in redundant build (if plugin works) > since it would be one Gradle build process. But it does do a full build if > you touch something at the root of

Re: Finer-grained test runs?

2020-07-09 Thread Robert Bradshaw
On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > >> If Brian's: it does not result in redundant build (if plugin works) since it >> would be one Gradle build process. But it does do a full build if you touch >> something at the root of the ancestry tree like core SDK or model. I would >> like

Re: Finer-grained test runs?

2020-07-09 Thread Kenneth Knowles
On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik wrote: > On Wed, Jul 8, 2020 at 9:22 PM Kenneth Knowles wrote: > >> I like your use of "ancestor" and "descendant". I will adopt it. >> >> On Wed, Jul 8, 2020 at 4:53 PM Robert Bradshaw >> wrote: >> >>> On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote:

Re: Finer-grained test runs?

2020-07-09 Thread Luke Cwik
On Wed, Jul 8, 2020 at 9:22 PM Kenneth Knowles wrote: > I like your use of "ancestor" and "descendant". I will adopt it. > > On Wed, Jul 8, 2020 at 4:53 PM Robert Bradshaw > wrote: > >> On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote: >> > >> > I'm not sure that breaking it up will be

Re: Finer-grained test runs?

2020-07-08 Thread Kenneth Knowles
I like your use of "ancestor" and "descendant". I will adopt it. On Wed, Jul 8, 2020 at 4:53 PM Robert Bradshaw wrote: > On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote: > > > > I'm not sure that breaking it up will be significantly faster since each > module needs to build its ancestors and

Re: Finer-grained test runs?

2020-07-08 Thread Robert Bradshaw
On Wed, Jul 8, 2020 at 4:44 PM Luke Cwik wrote: > > I'm not sure that breaking it up will be significantly faster since each > module needs to build its ancestors and run tests of itself and all of its > descendants which isn't a trivial amount of work. We have only so many > executors and

Re: Finer-grained test runs?

2020-07-08 Thread Luke Cwik
I'm not sure that breaking it up will be significantly faster since each module needs to build its ancestors and run tests of itself and all of its descendants which isn't a trivial amount of work. We have only so many executors and with the increased number of jobs, won't we just be waiting for

Re: Finer-grained test runs?

2020-07-08 Thread Kenneth Knowles
That's a good start. It is new enough and with few enough commits that I'd want to do some thorough experimentation. Our build is complex enough with a lot of ad hoc coding that we might end up maintaining whatever we choose... In my ideal scenario the list of "what else to test" would be

Re: Finer-grained test runs?

2020-07-08 Thread Brian Hulette
> We could have one "test the things" Jenkins job if the underlying tool (Gradle) could resolve what needs to be run. I think this would be much better. Otherwise it seems our Jenkins definitions are just duplicating information that's already stored in the build.gradle files which seems

Finer-grained test runs?

2020-07-08 Thread Kenneth Knowles
Hi all, I wanted to start a discussion about getting finer grained test execution more focused on particular artifacts/modules. In particular, I want to gather the downsides and impossibilities. So I will make a proposal that people can disagree with easily. Context: job_PreCommit_Java is a