Sorry wrong link: https://github.com/apache/beam/pull/9915
On Tue, Nov 26, 2019 at 5:12 PM Udi Meiri <eh...@google.com> wrote: > I looked at #9959 but it doesn't seem to modify setup.py? > The additional eggs for timeloop etc. are troubling though. Not sure where > those come from. > > On Tue, Nov 26, 2019 at 4:59 PM Chad Dombrova <chad...@gmail.com> wrote: > >> Is setup_requires being used somewhere else, because I'm still getting >> errors after removing it from sdks/python/setup.py. >> >> I removed it from this PR: https://github.com/apache/beam/pull/9959 >> >> Here's the gradle scan: >> https://scans.gradle.com/s/oinh5xpaly3dk/failure#top=0 >> >> The error shows up differently than before when >> setup_requries=['pytest-runner'] was present -- it's in a gradle traceback >> now rather than the console log. I've also seen different packages listed >> as the culprit (simplegeneric, timeloop). >> >> -chad >> >> >> >> On Tue, Nov 26, 2019 at 4:47 PM Udi Meiri <eh...@google.com> wrote: >> >>> Chad, I believe the answer is the "setup_requires" line is causing the >>> sdks/python/.eggs directory to be created. >>> >>> This command fails with the setup_requires line (same Errno 17), but >>> succeeds without it: >>> $ \rm -r .eggs/; ../../gradlew installGcpTest >>> [~8 failed tasks] >>> $ ls .eggs >>> pytest_runner-5.2-py2.7.egg pytest_runner-5.2-py3.5.egg >>> pytest_runner-5.2-py3.6.egg pytest_runner-5.2-py3.7.egg README.txt >>> >>> I'll go ahead and create a PR to remove setup_requires from setup.py. >>> >>> On Tue, Nov 26, 2019 at 4:16 PM Chad Dombrova <chad...@gmail.com> wrote: >>> >>>> It seems like the offending packages are those that only have source >>>> distributions (i.e. no wheels). But why are the eggs being installed in >>>> sdks/python/.eggs instead of into the virtualenv created by setupVirtualenv >>>> gradle task or by tox? >>>> >>>> >>>> On Tue, Nov 26, 2019 at 3:59 PM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> Basically, I believe what's happening is that a new Gradle task was >>>>> added that uses setup.py but doesn't have the same dependency on some main >>>>> setup.py task that all others depend on (list sdist). >>>>> >>>>> On Tue, Nov 26, 2019 at 3:49 PM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> Correction: the error is not gone after removing the line. I get >>>>>> instead: >>>>>> error: [Errno 17] File exists: >>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/dill-0.3.1.1-py2.7.egg' >>>>>> >>>>>> >>>>>> On Tue, Nov 26, 2019 at 3:45 PM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> I managed to recreate one of the issues with this command: >>>>>>> ~/src/beam/sdks/python$ \rm -r .eggs/ && for i in $(seq 2); do echo >>>>>>> "python setup.py -q nosetests --tests >>>>>>> apache_beam.pipeline_test:DoFnTest.test_incomparable_default &" | sh ; >>>>>>> done >>>>>>> >>>>>>> This reliably gives me: >>>>>>> OSError: [Errno 17] File exists: >>>>>>> '/usr/local/google/home/ehudm/src/beam/sdks/python/.eggs/pytest_runner-5.2-py2.7.egg' >>>>>>> >>>>>>> If I remove this line from setup.py the error is gone: >>>>>>> setup_requires=['pytest_runner'], >>>>>>> >>>>>>> >>>>>>> On Tue, Nov 26, 2019 at 2:54 PM Chad Dombrova <chad...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for looking into this. It seems like it might be something >>>>>>>> to do with data that is cached on the Jenkins slaves between runs, >>>>>>>> which >>>>>>>> may be what prevents this from showing up locally? >>>>>>>> >>>>>>>> If your theory about setuptools is correct, and it sounds likely, >>>>>>>> we should be able to lock down the version, which we should definitely >>>>>>>> be >>>>>>>> doing for all of our dependencies. >>>>>>>> >>>>>>>> -chad >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Nov 26, 2019 at 1:33 PM Ahmet Altay <al...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I tried to debug but did not make much progress. I cannot >>>>>>>>> reproduce locally, however all python precommits and postcommits are >>>>>>>>> failing. >>>>>>>>> >>>>>>>>> One guess is, setuptools released a new version that does not >>>>>>>>> support eggs a few days ago, that might be the cause ( >>>>>>>>> https://github.com/pypa/setuptools/blob/master/CHANGES.rst) but >>>>>>>>> that should have reproduced locally. >>>>>>>>> Maybe something is wrong with the jenkins machines, and we could >>>>>>>>> perhaps bring them to a clean state. >>>>>>>>> >>>>>>>>> I suspected this being related to pytest somehow (as the first 4 >>>>>>>>> JIRAs had pytest in the error line) but the error Chad saw is >>>>>>>>> different. >>>>>>>>> >>>>>>>>> +Valentyn Tymofieiev <valen...@google.com> and +Yifan Zou >>>>>>>>> <yifan...@google.com> could you help with looking into this? >>>>>>>>> >>>>>>>>> >>>>>>>>> Ahmet >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Nov 26, 2019 at 9:14 AM Luke Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I also started to see this on PRs that I'm reviewing. >>>>>>>>>> BEAM-8793, BEAM-8653, BEAM-8631, BEAM-8249 mention issues with >>>>>>>>>> setup.py and >>>>>>>>>> egg_info but this looks different then all of those so I filed >>>>>>>>>> BEAM-8831. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Nov 25, 2019 at 10:27 PM Chad Dombrova <chad...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Actually, it looks like I'm getting the same error on multiple >>>>>>>>>>> PRs: https://scans.gradle.com/s/ihfmrxr7evslw >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Nov 25, 2019 at 10:26 PM Chad Dombrova < >>>>>>>>>>> chad...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> The cython tests started failing on one of my PRs which were >>>>>>>>>>>> succeeding before. The error is one that I've never seen before >>>>>>>>>>>> (separated onto different lines to make it easier to read): >>>>>>>>>>>> >>>>>>>>>>>> Caused by: org.gradle.api.GradleException: >>>>>>>>>>>> Could not copy file >>>>>>>>>>>> >>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2 >>>>>>>>>>>> /src/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg' >>>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2 >>>>>>>>>>>> /src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/.eggs/simplegeneric-0.8.1-py2.7.egg'. >>>>>>>>>>>> >>>>>>>>>>>> Followed immediately by an error about could not create a >>>>>>>>>>>> directory of the same name. Here's the gradle scan: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://scans.gradle.com/s/ihfmrxr7evslw/failure?openFailures=WzFd&openStackTraces=WzZd#top=0 >>>>>>>>>>>> >>>>>>>>>>>> Any ideas? >>>>>>>>>>>> >>>>>>>>>>>> -chad >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>
