Thanks, updated the JIRA with a link to this thread and a note of what could be done.
On Mon, Apr 29, 2019 at 10:29 AM Udi Meiri <[email protected]> wrote: > Pip has a --cache-dir which should be safe with concurrent writes. > > On Fri, Apr 26, 2019 at 3:59 PM Ahmet Altay <[email protected]> wrote: > >> It is possible to download dependencies with pip to a local directory and >> install from there [1]. As a side benefit this is supposed to speed up the >> installation process. Since we setup virtualenv multiple times, this could >> actually help us in a single run. And if we can keep this cache across test >> runs we can reduce flakiness. >> >> [1] >> https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages >> >> On Fri, Apr 26, 2019 at 3:42 PM Valentyn Tymofieiev <[email protected]> >> wrote: >> >>> We do retry certain inherently flaky tests, for example, see[1]. This >>> practice should be used with caution, see discussion [2]. >>> >>> However retrying an individual test would not avoid the flake that Alex >>> brought up in this thread, we'd have to retry setupVirtualEnv task that is >>> executed once per suite of tests. Retrying just that task is different from >>> retrying the whole suite. >>> >>> [1] >>> https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41, >>> this was discussed >>> [2] >>> https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E >>> discussed >>> >>> On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri <[email protected]> wrote: >>> >>>> Alex, I changed my mind: I'm okay retrying single tests, just not >>>> entire suites of tests (e.g. if precommits take an hour, retrying the run >>>> takes up an additional hour on the Jenkins machine). >>>> This is more of an issue in Python, where gradle does not (currently) >>>> have insight into which tests failed and how to retry just them. >>>> >>>> >>>> >>>> On Fri, Apr 26, 2019 at 2:17 PM Alex Amato <[email protected]> wrote: >>>> >>>>> @Udi Meiri <[email protected]>, Is this true if the specific tests are >>>>> rerun? I don't think we should rerun all tests. >>>>> >>>>> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev < >>>>> [email protected]> wrote: >>>>> >>>>>> Preinstalling dependencies may affect the dependency resolution, and >>>>>> we may end up testing a different configuration than a user would have >>>>>> after installing beam into a clean environment. >>>>>> >>>>>> I do think pip uses cache, unless one specifies "--no-cache-dir". By >>>>>> default the cache is ~/.cache/pip. Looking up the log message in OP, we >>>>>> can >>>>>> see several "Using cached..." log entries. Not sure why futures was not >>>>>> fetched from cache or PyPi. Perhaps it is also a pip flake. >>>>>> >>>>>> I would be against wiping flakes under the rug by rerunning the whole >>>>>> suite after an error, but re-rerunning parts of the test environment set >>>>>> up, that are prone to environmental flakes, such as setupVirtualEnv seems >>>>>> reasonable. I agree with Udi that care should be taken to not overload >>>>>> Jenkins (e.g. retries should be limited) >>>>>> >>>>>
