Splitting into separate jobs that can be parallelized seems like a win for as long as Gradle task parallelization is disabled. Thanks for driving this improvement.
> I'm in favor of (simple!) build breaks going in before precommits finish, on the promise that the offending test(s) passed locally. +1. Some of our post-commits run even longer (many hours); it's not practical to wait for a full run to merge a fix. For targeted fixes/rollbacks to restore Jenkins test signal, I support using a passing local run as a stand-in for a full Jenkins execution. Note that if you run gradle with the --scan flag, it will produce a Build Scan URL to submit with you PR. On Thu, Oct 25, 2018 at 5:39 AM Kenneth Knowles <k...@apache.org> wrote: > The split did seemingly trim about 30 minutes off the Java precommit. Of > course the difference between 50 and 80 minutes won't qualitatively change > much. I don't see any other obvious and easy wins. I still like the split > for the separation of signals. > > Kenn > > On Tue, Oct 23, 2018 at 2:47 PM Robert Bradshaw <rober...@google.com> > wrote: > >> On Tue, Oct 23, 2018 at 11:28 PM Kenneth Knowles <k...@apache.org> wrote: >> >>> Hi all, >>> >>> Java Precommit duration is about 1h15. That is quite a burden. >>> Especially if something gets broken. >>> >> >> I'm in favor of (simple!) build breaks going in before precommits finish, >> on the promise that the offending test(s) passed locally. Short of that, we >> can roll back. >> >> If it were cheap to get a fast "this is probably good" signal, that could >> be useful as well, though once you hit the "I'm waiting long enough to go >> do something else" the difference between 20 minutes and 80 minutes is not >> that huge. >> >> >>> We turned off parallel builds, which we really need to re-enable. >>> >> >> +1 >> >> >>> But beyond that, I see low-hanging fruit that would most appropriately >>> be a separate Jenkins job. >>> >>> Here's a scan of a successful run: >>> https://scans.gradle.com/s/2s4bd5hc45wuy/timeline >>> >>> * 17m :beam-runners-google-cloud-dataflow-java-examples:preCommit >>> * 4m >>> :beam-runners-google-cloud-dataflow-java-examples-streaming:preCommit >>> These are integration tests that should have their own job & status >>> anyhow. We lumped them in because Maven can't do separate tests. Gradle >>> makes this cheap and easy. >>> >>> Then there are these which are the only other tasks over 1m: >>> >>> * 2m :beam-runners-google-cloud-dataflow-java-legacy-worker:test >>> * 2m :beam-runners-google-cloud-dataflow-java-fn-api-worker:test >>> * 2m :beam-sdks-java-nexmark:test >>> * 1m :beam-sdks-java-io-google-cloud-platform:test >>> * 1m :beam-sdks-java-io-hbase:test >>> * 1m :beam-sdks-java-extensions-sql:test >>> >>> Maybe not worth messing with these. Also if we remove all the shadowJar >>> and shadowTestJar tasks it actually looks like it would only save 5 >>> minutes, so I was hasty in thinking that would solve things. It will make >>> interactive work better (going from 30s to maybe <10s for rebuilds) but >>> won't help that much for Jenkins. >>> >>> Kenn >>> >> -- Got feedback? tinyurl.com/swegner-feedback
