Is anyone using gradle for daily development of python parts? Or is this just used as integration for Jenkins build? In case of the latter, clean isn't called anyway, right?
If the former, its probably worth to put more effort into python setup anyway, e.g. declaring proper input/outputs... If done correctly (if thats possible at all) clean will work out of the box anyway. On Wed, Feb 20, 2019 at 1:01 AM Udi Meiri <[email protected]> wrote: > I think I can solve this issue by removing the dependency and adding a > check to see if the virtualenv was created. > Otherwise, there shouldn't be anything to cleanup anyway. > > On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <[email protected]> > wrote: > >> Thanks Michael, your assessment was correct. I needed python3.5 on my >> $PATH. >> >> For completeness, I needed this >> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib> >> to get pyenv to install python 3.5.6 on macOS: >> >> ``` >> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6 >> ``` >> >> `./gradlew clean` worked after that. >> >> >> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <[email protected]> >> wrote: >> >>> As far as I understand, the build got bound to 3.5 [1]. >>> >>> Could it be that you do not have python3.5 on your path? e.g. try python3.5 >>> --version >>> >>> If that is missing, you will not be able to run any py3 task, I guess... >>> >>> So, to get out of this state, you have to get python3.5 command working. >>> >>> At least on my machine, './gradlew clean' is working iff python3.5 is on >>> my path. >>> >>> michel >>> >>> [1] >>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591 >>> >>> >>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <[email protected]> >>> wrote: >>> >>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on >>>> two py3 setupVirtualEnv tasks, >>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and >>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's >>>> full output >>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39> >>>> . >>>> >>>> Any tips how to get out of this state, or what is causing it? >>>> >>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <[email protected]> wrote: >>>> >>>>> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459 >>>>> . >>>>> >>>>> Kenn >>>>> >>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <[email protected]> wrote: >>>>> >>>>>> Probably related, a top level ./gradlew clean fails with the >>>>>> following: >>>>>> >>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED >>>>>> The path python3.5 (from --python=python3.5) does not exist >>>>>> >>>>>> Can we just limit clean to do cleanup?! >>>>>> >>>>>> >>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Just some background, grpcio-tools is what's used to do the proto >>>>>>> generation. Unfortunately it's expensive to compile and doesn't >>>>>>> provide very many wheels, so we want to install it once, not every >>>>>>> time. (It's also used in more than just tests; one needs it every >>>>>>> time >>>>>>> the .proto files change.) >>>>>>> >>>>>>> That being said, we could probably do a much cheaper clean. >>>>>>> >>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <[email protected]> wrote: >>>>>>> > >>>>>>> > grpcio-tools could probably be moved under the "test" tag in >>>>>>> setup.py. Not sure why it has to be specified in gradle configs. >>>>>>> > >>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <[email protected]> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean` >>>>>>> without installing gcpio-tools, etc? >>>>>>> >> >>>>>>> >> Kenn >>>>>>> >> >>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <[email protected]> >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>> setup.py has requirements like setuptools, which are installed >>>>>>> in the virtual environment. >>>>>>> >>> So even running the clean command requires the virtualenv to be >>>>>>> set up. >>>>>>> >>> >>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if >>>>>>> setupVirtualenv has not been run. (perhaps by checking for the >>>>>>> existence of >>>>>>> its output directory) >>>>>>> >>> >>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <[email protected]> >>>>>>> wrote: >>>>>>> >>>> >>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to >>>>>>> record the conclusion. Doesn't require Beam knowledge so I labeled >>>>>>> "starter". >>>>>>> >>>> >>>>>>> >>>> Kenn >>>>>>> >>>> >>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey < >>>>>>> [email protected]> wrote: >>>>>>> >>>>> >>>>>>> >>>>> This seems to be on purpose [1] >>>>>>> >>>>> >>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We >>>>>>> probably should work around that. >>>>>>> >>>>> >>>>>>> >>>>> [1] >>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610 >>>>>>> >>>>> >>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean` >>>>>>> fails due to failure of `setupVirtualEnv` tasks. >>>>>>> >>>>>> >>>>>>> >>>>>> Manu Zhang >>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles < >>>>>>> [email protected]>, wrote: >>>>>>> >>>>>> >>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` >>>>>>> tasks that invoke things such as `setup.py bdist_wheel for >>>>>>> grpcio-tools`. >>>>>>> Overall it took 4 minutes. Is this intended? >>>>>>> >>>>>> >>>>>>> >>>>>> Kenn >>>>>>> >>>>>>
