On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri <eh...@google.com> wrote: > I have given this some thought honestly don't know if splitting into > separate jobs will help. > - I have seen race conditions with running setuptools in parallel, so more > isolation is better. OTOH, if 2 setuptools scripts run at the same time on > the same machine they might still race (same homedir, same /tmp dir). > - Retrying due to flakes will be faster, but if something is broken you'll > need to write 4x the number of "run python precommit" phrases. >
there is also "retest this please" which will rerun all precommits. > - Increased parallelism may also run into quota issues with Dataflow. > > What benefits do you see from splitting up the jobs? > > On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova <chad...@gmail.com> wrote: > >> After this PR goes in should we revisit breaking up the python tests into >> separate jenkins jobs by python version? One of the problems with that >> plan originally was that we lost the parallelism that gradle provides >> because we were left with only one tox task per jenkins job, and so the >> total time to complete all python jenkins jobs went up a lot. With >> pytest + xdist we should hopefully be able to keep the parallelism even >> with just one tox task. This could be a big win. I feel like I'm spending >> more time monitoring and re-queuing timed-out jenkins jobs lately than I am >> writing code. >> >> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri <eh...@google.com> wrote: >> >>> This PR <https://github.com/apache/beam/pull/10322> (in review) >>> migrates py27-gcp to using pytest. >>> It reduces the testPy2Gcp task down to ~13m >>> <https://scans.gradle.com/s/kj7ogemnd3toe/timeline?details=ancsbov425524> >>> (from ~45m). This speedup will probably be lower once all 8 tasks are using >>> pytest. >>> It also adds 5 previously uncollected tests. >>> >>
