Thank you Valentyn! On Wed, Feb 12, 2020 at 11:32 AM Valentyn Tymofieiev <valen...@google.com> wrote:
> Yes, otherwise all Python tests will continue to fail until Avro comes up > with a new release. Sent: https://github.com/apache/beam/pull/10844 > > On Wed, Feb 12, 2020 at 11:08 AM Ahmet Altay <al...@google.com> wrote: > >> Should we update Beam's setup.py to skip this avro-python3 version? >> >> On Wed, Feb 12, 2020 at 10:57 AM Alan Krumholz <alan.krumh...@betterup.co> >> wrote: >> >>> makes sense. I'll add this workaround for now. >>> Thanks so much for your help! >>> >>> On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> Alan, Dataflow workers preinstall Beam SDK dependencies, including (a >>>> working version) of avro-python3. So after reading your email once again, I >>>> think in your case you were not able to install Beam SDK locally. So a >>>> workaround for you would be to `pip install avro-python3==1.9.1` or `pip >>>> install pycodestyle` before installing Beam, until AVRO-2737 is resolved. >>>> >>>> >>>> On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and >>>>> it received attention. >>>>> >>>>> On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> Opened https://issues.apache.org/jira/browse/AVRO-2738 >>>>>> >>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>>>> valen...@google.com> wrote: >>>>>> >>>>>>> Here's a short repro: >>>>>>> >>>>>>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch >>>>>>> root@04b45a100d16:/# pip install avro-python3 >>>>>>> Collecting avro-python3 >>>>>>> Downloading avro-python3-1.9.2.tar.gz (37 kB) >>>>>>> ERROR: Command errored out with exit status 1: >>>>>>> command: /usr/local/bin/python -c 'import sys, setuptools, >>>>>>> tokenize; sys.argv[0] = >>>>>>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"'; >>>>>>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize, >>>>>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>>>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>>>>>> egg_info --egg-base /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info >>>>>>> cwd: /tmp/pip-install-mmy4vspt/avro-python3/ >>>>>>> Complete output (5 lines): >>>>>>> Traceback (most recent call last): >>>>>>> File "<string>", line 1, in <module> >>>>>>> File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py", line >>>>>>> 41, in <module> >>>>>>> import pycodestyle >>>>>>> ModuleNotFoundError: No module named 'pycodestyle' >>>>>>> ---------------------------------------- >>>>>>> ERROR: Command errored out with exit status 1: python setup.py >>>>>>> egg_info Check the logs for full command output. >>>>>>> root@04b45a100d16:/# >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>>> Yes, it is a bug in the recent Avro release. We should report it >>>>>>>> to the Avro maintainers. The workaround is to downgrade avro-python3 to >>>>>>>> 1.9.1, for example via requirements.txt. >>>>>>>> >>>>>>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz <sniem...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and >>>>>>>>> added pycodestyle as a dependency, probably related? >>>>>>>>> >>>>>>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <lc...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +dev <dev@beam.apache.org> >>>>>>>>>> >>>>>>>>>> There was recently an update to add autoformatting to the Python >>>>>>>>>> SDK[1]. >>>>>>>>>> >>>>>>>>>> I'm seeing this during testing of a PR as well. >>>>>>>>>> >>>>>>>>>> 1: >>>>>>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E >>>>>>>>>> >>>>>>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz < >>>>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>>>> >>>>>>>>>>> Some more information for this as I still can't get to fix it.... >>>>>>>>>>> >>>>>>>>>>> This job is triggered using the beam[gcp] python sdk from a >>>>>>>>>>> KubeFlow Pipelines component which runs on top of docker image: >>>>>>>>>>> tensorflow/tensorflow:1.13.1-py3 >>>>>>>>>>> >>>>>>>>>>> I just checked and that image hasn't been updated recently. I >>>>>>>>>>> also redeployed my pipeline to another (older) deployment of KFP >>>>>>>>>>> and it >>>>>>>>>>> gives me the same error (which tells me this isn't an internal KFP >>>>>>>>>>> problem) >>>>>>>>>>> >>>>>>>>>>> The exact same pipeline/code running on the exact same image has >>>>>>>>>>> been running fine for days. Did anything changed on the >>>>>>>>>>> beam/dataflow side >>>>>>>>>>> since yesterday morning? >>>>>>>>>>> >>>>>>>>>>> Thanks for your help! this is a production pipeline that is not >>>>>>>>>>> running for us :( >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz < >>>>>>>>>>> alan.krumh...@betterup.co> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, I have a scheduled daily job that I have been running fine >>>>>>>>>>>> in dataflow for days now. >>>>>>>>>>>> We haven't changed anything on this code but this morning run >>>>>>>>>>>> failed (it couldn't even spin up the job) >>>>>>>>>>>> The job submits a setup.py file (that also hasn't changed) but >>>>>>>>>>>> maybe is causing the problem? (based on the error I'm getting) >>>>>>>>>>>> >>>>>>>>>>>> Anyone else having the same issue? or know how to fix it? >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> ERROR: Complete output from command python setup.py egg_info: >>>>>>>>>>>> 2 ERROR: Traceback (most recent call last): >>>>>>>>>>>> 3 File "<string>", line 1, in <module> >>>>>>>>>>>> 4 File "/tmp/pip-install-42zyi89t/avro-python3/setup.py", line >>>>>>>>>>>> 41, in <module> >>>>>>>>>>>> 5 import pycodestyle >>>>>>>>>>>> 6 ImportError: No module named 'pycodestyle' >>>>>>>>>>>> 7 ---------------------------------------- >>>>>>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>>>> code 1 in /tmp/pip-install-42zyi89t/avro-python3/ >>>>>>>>>>>> 9 ERROR: Complete output from command python setup.py egg_info: >>>>>>>>>>>> 10 ERROR: Traceback (most recent call last): >>>>>>>>>>>> 11 File "<string>", line 1, in <module> >>>>>>>>>>>> 12 File "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", >>>>>>>>>>>> line 41, in <module> >>>>>>>>>>>> 13 import pycodestyle >>>>>>>>>>>> 14 ImportError: No module named 'pycodestyle' >>>>>>>>>>>> 15 ---------------------------------------- >>>>>>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>>>> code 1 in /tmp/pip-install-wrqytf9a/avro-python3/ >>>>>>>>>>>> >>>>>>>>>>>
