Re: cython test instability

2019-11-26 Thread Chad Dombrova
yeah, I've excised both test_requires and setup_requires in my test simplification PR: https://github.com/apache/beam/pull/10038 I'm happy to see those go sooner rather than later, as it'll reduce the scope of my PR. The rest of my PR is about ensuring that build dependencies like cython and

Re: cython test instability

2019-11-26 Thread Udi Meiri
I'm not sure about where the error with the simplegeneric, timeloop .eggs directories come from, but I did figure out that they don't get installed as eggs if you add them to the "test" extras in setup.py, e.g.: extras_require={ 'docs': ['Sphinx>=1.5.2,<2.0'], 'test':

Re: cython test instability

2019-11-26 Thread Chad Dombrova
Sorry wrong link: https://github.com/apache/beam/pull/9915 On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri wrote: > I looked at #9959 but it doesn't seem to modify setup.py? > The additional eggs for timeloop etc. are troubling though. Not sure where > those come from. > > On Tue, Nov 26, 2019 at

Re: cython test instability

2019-11-26 Thread Udi Meiri
I looked at #9959 but it doesn't seem to modify setup.py? The additional eggs for timeloop etc. are troubling though. Not sure where those come from. On Tue, Nov 26, 2019 at 4:59 PM Chad Dombrova wrote: > Is setup_requires being used somewhere else, because I'm still getting > errors after

Re: cython test instability

2019-11-26 Thread Chad Dombrova
Is setup_requires being used somewhere else, because I'm still getting errors after removing it from sdks/python/setup.py. I removed it from this PR: https://github.com/apache/beam/pull/9959 Here's the gradle scan: https://scans.gradle.com/s/oinh5xpaly3dk/failure#top=0 The error shows up

Re: cython test instability

2019-11-26 Thread Udi Meiri
Chad, I believe the answer is the "setup_requires" line is causing the sdks/python/.eggs directory to be created. This command fails with the setup_requires line (same Errno 17), but succeeds without it: $ \rm -r .eggs/; ../../gradlew installGcpTest [~8 failed tasks] $ ls .eggs

Re: cython test instability

2019-11-26 Thread Chad Dombrova
It seems like the offending packages are those that only have source distributions (i.e. no wheels). But why are the eggs being installed in sdks/python/.eggs instead of into the virtualenv created by setupVirtualenv gradle task or by tox? On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri wrote: >

Re: cython test instability

2019-11-26 Thread Udi Meiri
Basically, I believe what's happening is that a new Gradle task was added that uses setup.py but doesn't have the same dependency on some main setup.py task that all others depend on (list sdist). On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri wrote: > Correction: the error is not gone after

Re: cython test instability

2019-11-26 Thread Udi Meiri
Correction: the error is not gone after removing the line. I get instead: error: [Errno 17] File exists: '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg' On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri wrote: > I managed to recreate one of the issues with this

Re: cython test instability

2019-11-26 Thread Udi Meiri
I managed to recreate one of the issues with this command: ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo "python setup.py -q nosetests --tests apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ; done This reliably gives me: OSError: [Errno 17] File exists:

Re: cython test instability

2019-11-26 Thread Chad Dombrova
Thanks for looking into this. It seems like it might be something to do with data that is cached on the Jenkins slaves between runs, which may be what prevents this from showing up locally? If your theory about setuptools is correct, and it sounds likely, we should be able to lock down the

Re: cython test instability

2019-11-26 Thread Ahmet Altay
I tried to debug but did not make much progress. I cannot reproduce locally, however all python precommits and postcommits are failing. One guess is, setuptools released a new version that does not support eggs a few days ago, that might be the cause (

Re: cython test instability

2019-11-26 Thread Luke Cwik
I also started to see this on PRs that I'm reviewing. BEAM-8793, BEAM-8653, BEAM-8631, BEAM-8249 mention issues with setup.py and egg_info but this looks different then all of those so I filed BEAM-8831. On Mon, Nov 25, 2019 at 10:27 PM Chad Dombrova wrote: > Actually, it looks like I'm

Re: cython test instability

2019-11-25 Thread Chad Dombrova
Actually, it looks like I'm getting the same error on multiple PRs: https://scans.gradle.com/s/ihfmrxr7evslw On Mon, Nov 25, 2019 at 10:26 PM Chad Dombrova wrote: > Hi all, > The cython tests started failing on one of my PRs which were succeeding > before. The error is one that I've never

cython test instability

2019-11-25 Thread Chad Dombrova
Hi all, The cython tests started failing on one of my PRs which were succeeding before. The error is one that I've never seen before (separated onto different lines to make it easier to read): Caused by: org.gradle.api.GradleException: Could not copy file