On Thu, Jul 9, 2020 at 8:40 AM Luke Cwik <lc...@google.com> wrote: > >> If Brian's: it does not result in redundant build (if plugin works) since it >> would be one Gradle build process. But it does do a full build if you touch >> something at the root of the ancestry tree like core SDK or model. I would >> like to avoid automatically testing descendants if we can, since things like >> Nexmark and most IOs are not sensitive to the vast majority of model or core >> SDK changes. Runners are borderline. > > I believe that the cost of fixing an issue that is found later once the test > starts failing because the test wasn't run as part of the PR has a much > higher order of magnitude of cost to triage and fix. Mostly due to loss of > context from the PR author/reviewer and if the culprit PR can't be found then > whoever is trying to fix it.
Huge +1 to this. Ideally we could count on the build system (and good caching) to only test what actually needs to be tested, and with work being done on runners and IOs this would be a small subset of our entire suite. When working lower in the stack (and I am prone to do) I think it's acceptable to have longer wait times--and would *much* rather pay that price than discover things later. Perhaps some things could be surgically removed (it would be interesting to mine data on how often test failures in the "leaves" catch real issues), but I would do that with care. That being said, flakiness is really an issues (and it seems these days I have to re-run tests, often multiple times, to get a PR to green; splitting up jobs could help that as well).
