I think it makes sense to remove some of the extra FnApiRunner configurations. Perhaps some of the multiworkers and some of the grpc versions? Best -P.
On Fri, Oct 25, 2019 at 12:27 PM Robert Bradshaw <rober...@google.com> wrote: > It looks like fn_api_runner_test.py is quite expensive, taking 10-15+ > minutes on each version of Python. This test consists of a base class > that is basically a validates runner suite, and is then run in several > configurations, many more of which (including some expensive ones) > have been added lately. > > class FnApiRunnerTest(unittest.TestCase): > class FnApiRunnerTestWithGrpc(FnApiRunnerTest): > class FnApiRunnerTestWithGrpcMultiThreaded(FnApiRunnerTest): > class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest): > class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest): > class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest): > class FnApiRunnerTestWithBundleRepeat(FnApiRunnerTest): > class FnApiRunnerTestWithBundleRepeatAndMultiWorkers(FnApiRunnerTest): > > I'm not convinced we need to run all of these permutations, or at > least not all tests in all permutations. > > On Fri, Oct 25, 2019 at 10:57 AM Valentyn Tymofieiev > <valen...@google.com> wrote: > > > > I took another look at this and precommit ITs are already running in > parallel, albeit in the same suite. However it appears Python precommits > became slower, especially Python 2 precommits [35 min per suite x 3 > suites], see [1]. Not sure yet what caused the increase, but precommits > used to be faster. Perhaps we have added a slow test or a lot of new tests. > > > > [1] https://scans.gradle.com/s/jvcw5fpqfc64k/timeline?task=ancsbov425524 > > > > On Thu, Oct 24, 2019 at 4:53 PM Ahmet Altay <al...@google.com> wrote: > >> > >> Ack. Separating precommit ITs to a different suite sounds good. Anyone > is interested in doing that? > >> > >> On Thu, Oct 24, 2019 at 2:41 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>> > >>> This should not increase the queue time substantially, since precommit > ITs are running sequentially with precommit tests, unlike multiple > precommit tests which run in parallel to each other. > >>> > >>> The precommit ITs we run are batch and streaming wordcount tests on > Py2 and one Py3 version, so it's not a lot of tests. > >>> > >>> On Thu, Oct 24, 2019 at 1:07 PM Ahmet Altay <al...@google.com> wrote: > >>>> > >>>> +1 to separating ITs from precommit. Downside would be, when Chad > tried to do something similar [1] it was noted that the total time to run > all precommit tests would increase and also potentially increase the queue > time. > >>>> > >>>> Another alternative, we could run a smaller set of IT tests in > precommits and run the whole suite as part of post commit tests. > >>>> > >>>> [1] https://github.com/apache/beam/pull/9642 > >>>> > >>>> On Thu, Oct 24, 2019 at 12:15 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>> > >>>>> One improvement could be move to Precommit IT tests into a separate > suite from precommit tests, and run it in parallel. > >>>>> > >>>>> On Thu, Oct 24, 2019 at 11:41 AM Brian Hulette <bhule...@google.com> > wrote: > >>>>>> > >>>>>> Python Precommits are taking quite a while now [1]. Just visually > it looks like the average length is 1.5h or so, but it spikes up to 2h. > I've had several precommit runs get aborted due to the 2 hour limit. > >>>>>> > >>>>>> It looks like there was a spike up above 1h back on 9/6 and the > duration has been steadily rising since then. Is there anything we can do > about this? > >>>>>> > >>>>>> Brian > >>>>>> > >>>>>> [1] > http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1&from=now-90d&to=now&fullscreen&panelId=4 >
