Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Thanks, updated the JIRA with a link to this thread and a note of what could be done. On Mon, Apr 29, 2019 at 10:29 AM Udi Meiri wrote: > Pip has a --cache-dir which should be safe with concurrent writes. > > On Fri, Apr 26, 2019 at 3:59 PM Ahmet Altay wrote: > >> It is possible to download

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Pip has a --cache-dir which should be safe with concurrent writes. On Fri, Apr 26, 2019 at 3:59 PM Ahmet Altay wrote: > It is possible to download dependencies with pip to a local directory and > install from there [1]. As a side benefit this is supposed to speed up the > installation process.

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

It is possible to download dependencies with pip to a local directory and install from there [1]. As a side benefit this is supposed to speed up the installation process. Since we setup virtualenv multiple times, this could actually help us in a single run. And if we can keep this cache across

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

We do retry certain inherently flaky tests, for example, see[1]. This practice should be used with caution, see discussion [2]. However retrying an individual test would not avoid the flake that Alex brought up in this thread, we'd have to retry setupVirtualEnv task that is executed once per

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Alex, I changed my mind: I'm okay retrying single tests, just not entire suites of tests (e.g. if precommits take an hour, retrying the run takes up an additional hour on the Jenkins machine). This is more of an issue in Python, where gradle does not (currently) have insight into which tests

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

@Udi Meiri , Is this true if the specific tests are rerun? I don't think we should rerun all tests. On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev wrote: > Preinstalling dependencies may affect the dependency resolution, and we > may end up testing a different configuration than a user

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Preinstalling dependencies may affect the dependency resolution, and we may end up testing a different configuration than a user would have after installing beam into a clean environment. I do think pip uses cache, unless one specifies "--no-cache-dir". By default the cache is ~/.cache/pip.

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Good points, Kenn : ) - I know Yifan looked into running tests using Docker. I imagine a testing container could have a set of dependencies installed, and reduce the amount of installations we perform from PyPi. Best -P. On Fri, Apr 26, 2019 at 11:49 AM Kenneth Knowles wrote: > Flakiness in

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Flakiness in Java got a lot better when we put the Maven cache outside the wiped build directory. I am not sure about Gradle now... It is obviously less hermetic, but these things should be immutable so a cache is acceptable. Is there a way to achieve this for Python? For Maven/Gradle a package

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

That runs the risk of overloading our test machines when the build goes red. On Fri, Apr 26, 2019 at 11:29 AM Alex Amato wrote: > It would be ideal to not need manual steps. If known flakey tests can be > auto retried that would be a great improvement. > > On Fri, Apr 26, 2019 at 11:24 AM

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

It would be ideal to not need manual steps. If known flakey tests can be auto retried that would be a great improvement. On Fri, Apr 26, 2019 at 11:24 AM Valentyn Tymofieiev wrote: > We could do something along the lines of retry with a back-off. Note that > Java tests also have this problem as

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

We could do something along the lines of retry with a back-off. Note that Java tests also have this problem as we sometimes fail to fetch packages from Maven Central. On Fri, Apr 26, 2019 at 11:19 AM Pablo Estrada wrote: > hm no, these are somewhat common. Yes, I think we could have retries to

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

No, just re-run the tests. For the record, there are PyPi error-rates in https://status.python.org/#system-metrics. On Fri, Apr 26, 2019 at 11:15 AM Alex Amato wrote: > Okay but this occurred on jenkins. So does the machine need an update? > > On Fri, Apr 26, 2019 at 10:43 AM Valentyn

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

hm no, these are somewhat common. Yes, I think we could have retries to try to fix this sort of problem. Perhaps a mixture of reusing a virtualenv, and having retries when creating it? On Fri, Apr 26, 2019 at 11:15 AM Alex Amato wrote: > Okay but this occurred on jenkins. So does the machine

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Okay but this occurred on jenkins. So does the machine need an update? On Fri, Apr 26, 2019 at 10:43 AM Valentyn Tymofieiev wrote: > I think you hit a pypi flake. > > pip install futures>=2.2.0 works fine for me. > > On Fri, Apr 26, 2019 at 9:41 AM Alex Amato wrote: > >> Would be nice to fix

Re: [BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

I think you hit a pypi flake. pip install futures>=2.2.0 works fine for me. On Fri, Apr 26, 2019 at 9:41 AM Alex Amato wrote: > Would be nice to fix this as it can slow down PRs. I am not sure if this one > is fixed on retry yet or not. > > > >

[BEAM-7164] Python precommit failing on Java PRs. dataflow:setupVirtualenv

Would be nice to fix this as it can slow down PRs. I am not sure if this one is fixed on retry yet or not. *https://issues.apache.org/jira/browse/BEAM-7164?filter=-2 *