Good proposal. I think it should be considered in tandem with the "No commit on red post-commit" proposal and could be far more ambitious than 2 hours. For example, something in the <15-20 mins range, say, would be much less of an inconvenience to the development effort. Go takes ~3 mins, which means that it is practical to wait until a PR is green before asking anyone to look at it. If I need to wait for a Java or Python pre-commit, I task switch and come back later. If the post-commits are enforced to be green, we could possibly gain a much more productive flow at the cost of the occasional post-commit break, compared to now. Maybe IOs can be less extensively tested pre-commit, for example, or only if actually changed?
I also like Robert's suggestion of spitting up pre-commits into something more fine-grained to get a clear partial signal quicker. If we have an adequate number of Jenkins slots, it might also speed things up overall. Thanks, Henning On Fri, May 18, 2018 at 12:30 PM Scott Wegner <sweg...@google.com> wrote: > re: intelligently skipping tests for code that doesn't change (i.e. Java > tests on Python PR): this should be possible. We already have build-caching > enabled in Gradle, but I believe it is local to the git workspace and > doesn't persist between Jenkins runs. > > With a quick search, I see there is a Jenkins Build Cacher Plugin [1] that > hooks into Gradle build cache and does exactly what we need. Does anybody > know whether we could get this enabled on our Jenkins? > > [1] https://wiki.jenkins.io/display/JENKINS/Job+Cacher+Plugin > > On Fri, May 18, 2018 at 12:08 PM Robert Bradshaw <rober...@google.com> > wrote: > >> [somehow my email got garbled...] >> >> Now that we're using gradle, perhaps we could be more intelligent about >> only running the affected tests? E.g. when you touch Python (or Go) you >> shouldn't need to run the Java precommit at all, which would reduce the >> latency for those PRs and also the time spent in queue. Presumably this >> could even be applied per-module for the Java tests. (Maybe a large, shared >> build cache could help here as well...) >> >> I also wouldn't be opposed to a quicker immediate signal, plus more >> extensive tests before actually merging. It's also nice to not have to wait >> an hour to see that you have a lint error; quick stuff like that could be >> signaled quickly before a contributor looses context. >> >> - Robert >> >> >> >> On Fri, May 18, 2018 at 5:55 AM Kenneth Knowles <k...@google.com> wrote: >> >>> I like the idea. I think it is a good time for the project to start >>> tracking this and keeping it usable. >>> >>> Certainly 2 hours is more than enough, is that not so? The Java >>> precommit seems to take <=40 minutes while Python takes ~20 and Go is so >>> fast it doesn't matter. Do we have enough stragglers that we don't make >>> it in the 95th percentile? Is the time spent in the Jenkins queue? >>> >>> For our current coverage, I'd be willing to go for: >>> >>> - 1 hr hard cap (someone better at stats could choose %ile) >>> - roll back or remove test from precommit if fix looks like more than 1 >>> week (roll back if it is perf degradation, remove test from precommit if it >>> is additional coverage that just doesn't fit in the time) >>> >>> There's a longer-term issue that doing a full build each time is >>> expected to linearly scale up with the size of our repo (it is the monorepo >>> problem but for a minirepo) so there is no cap that is feasible until we >>> have effective cross-build caching. And my long-term goal would be <30 >>> minutes. At the latency of opening a pull request and then checking your >>> email that's not burdensome, but an hour is. >>> >>> Kenn >>> >>> On Thu, May 17, 2018 at 6:54 PM Udi Meiri <eh...@google.com> wrote: >>> >>>> HI, >>>> I have a proposal to improve contributor experience by keeping >>>> precommit times low. >>>> >>>> I'm looking to get community consensus and approval about: >>>> 1. How long should precommits take. 2 hours @95th percentile over the >>>> past 4 weeks is the current proposal. >>>> 2. The process for dealing with slowness. Do we: fix, roll back, remove >>>> a test from precommit? >>>> Rolling back if a fix is estimated to take longer than 2 weeks is the >>>> current proposal. >>>> >>>> >>>> https://docs.google.com/document/d/1udtvggmS2LTMmdwjEtZCcUQy6aQAiYTI3OrTP8CLfJM/edit?usp=sharing >>>> >>>
