Related: we have dependencies on avro, avro-python3, and fastavro. fastavro supports both python 2 and 3. Could we reduce this dependency list and depend only on fastavro? If we need avro and avro-python3 for the purposes of testing only, we can move them to test only dependencies.
+Chamikara Jayalath <[email protected]>, because I vaguely remember him working on this. The reason I am calling for this is the impact of bad dependency releases are high. All previously released Beam versions will be impacted. Reducing the dependency list will reduce the risk. Ahmet On Wed, Feb 12, 2020 at 12:02 PM Ahmet Altay <[email protected]> wrote: > Thank you Valentyn! > > On Wed, Feb 12, 2020 at 11:32 AM Valentyn Tymofieiev <[email protected]> > wrote: > >> Yes, otherwise all Python tests will continue to fail until Avro comes up >> with a new release. Sent: https://github.com/apache/beam/pull/10844 >> >> On Wed, Feb 12, 2020 at 11:08 AM Ahmet Altay <[email protected]> wrote: >> >>> Should we update Beam's setup.py to skip this avro-python3 version? >>> >>> On Wed, Feb 12, 2020 at 10:57 AM Alan Krumholz < >>> [email protected]> wrote: >>> >>>> makes sense. I'll add this workaround for now. >>>> Thanks so much for your help! >>>> >>>> On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev < >>>> [email protected]> wrote: >>>> >>>>> Alan, Dataflow workers preinstall Beam SDK dependencies, including (a >>>>> working version) of avro-python3. So after reading your email once again, >>>>> I >>>>> think in your case you were not able to install Beam SDK locally. So a >>>>> workaround for you would be to `pip install avro-python3==1.9.1` or `pip >>>>> install pycodestyle` before installing Beam, until AVRO-2737 is resolved. >>>>> >>>>> >>>>> On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev < >>>>> [email protected]> wrote: >>>>> >>>>>> Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and >>>>>> it received attention. >>>>>> >>>>>> On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Opened https://issues.apache.org/jira/browse/AVRO-2738 >>>>>>> >>>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Here's a short repro: >>>>>>>> >>>>>>>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch >>>>>>>> root@04b45a100d16:/# pip install avro-python3 >>>>>>>> Collecting avro-python3 >>>>>>>> Downloading avro-python3-1.9.2.tar.gz (37 kB) >>>>>>>> ERROR: Command errored out with exit status 1: >>>>>>>> command: /usr/local/bin/python -c 'import sys, setuptools, >>>>>>>> tokenize; sys.argv[0] = >>>>>>>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"'; >>>>>>>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize, >>>>>>>> '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', >>>>>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' >>>>>>>> egg_info --egg-base /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info >>>>>>>> cwd: /tmp/pip-install-mmy4vspt/avro-python3/ >>>>>>>> Complete output (5 lines): >>>>>>>> Traceback (most recent call last): >>>>>>>> File "<string>", line 1, in <module> >>>>>>>> File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py", line >>>>>>>> 41, in <module> >>>>>>>> import pycodestyle >>>>>>>> ModuleNotFoundError: No module named 'pycodestyle' >>>>>>>> ---------------------------------------- >>>>>>>> ERROR: Command errored out with exit status 1: python setup.py >>>>>>>> egg_info Check the logs for full command output. >>>>>>>> root@04b45a100d16:/# >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Yes, it is a bug in the recent Avro release. We should report it >>>>>>>>> to the Avro maintainers. The workaround is to downgrade avro-python3 >>>>>>>>> to >>>>>>>>> 1.9.1, for example via requirements.txt. >>>>>>>>> >>>>>>>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and >>>>>>>>>> added pycodestyle as a dependency, probably related? >>>>>>>>>> >>>>>>>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +dev <[email protected]> >>>>>>>>>>> >>>>>>>>>>> There was recently an update to add autoformatting to the Python >>>>>>>>>>> SDK[1]. >>>>>>>>>>> >>>>>>>>>>> I'm seeing this during testing of a PR as well. >>>>>>>>>>> >>>>>>>>>>> 1: >>>>>>>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E >>>>>>>>>>> >>>>>>>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Some more information for this as I still can't get to fix >>>>>>>>>>>> it.... >>>>>>>>>>>> >>>>>>>>>>>> This job is triggered using the beam[gcp] python sdk from a >>>>>>>>>>>> KubeFlow Pipelines component which runs on top of docker image: >>>>>>>>>>>> tensorflow/tensorflow:1.13.1-py3 >>>>>>>>>>>> >>>>>>>>>>>> I just checked and that image hasn't been updated recently. I >>>>>>>>>>>> also redeployed my pipeline to another (older) deployment of KFP >>>>>>>>>>>> and it >>>>>>>>>>>> gives me the same error (which tells me this isn't an internal KFP >>>>>>>>>>>> problem) >>>>>>>>>>>> >>>>>>>>>>>> The exact same pipeline/code running on the exact same image >>>>>>>>>>>> has been running fine for days. Did anything changed on the >>>>>>>>>>>> beam/dataflow >>>>>>>>>>>> side since yesterday morning? >>>>>>>>>>>> >>>>>>>>>>>> Thanks for your help! this is a production pipeline that is not >>>>>>>>>>>> running for us :( >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, I have a scheduled daily job that I have been running fine >>>>>>>>>>>>> in dataflow for days now. >>>>>>>>>>>>> We haven't changed anything on this code but this morning run >>>>>>>>>>>>> failed (it couldn't even spin up the job) >>>>>>>>>>>>> The job submits a setup.py file (that also hasn't changed) but >>>>>>>>>>>>> maybe is causing the problem? (based on the error I'm getting) >>>>>>>>>>>>> >>>>>>>>>>>>> Anyone else having the same issue? or know how to fix it? >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> ERROR: Complete output from command python setup.py egg_info: >>>>>>>>>>>>> 2 ERROR: Traceback (most recent call last): >>>>>>>>>>>>> 3 File "<string>", line 1, in <module> >>>>>>>>>>>>> 4 File "/tmp/pip-install-42zyi89t/avro-python3/setup.py", >>>>>>>>>>>>> line 41, in <module> >>>>>>>>>>>>> 5 import pycodestyle >>>>>>>>>>>>> 6 ImportError: No module named 'pycodestyle' >>>>>>>>>>>>> 7 ---------------------------------------- >>>>>>>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>>>>> code 1 in /tmp/pip-install-42zyi89t/avro-python3/ >>>>>>>>>>>>> 9 ERROR: Complete output from command python setup.py >>>>>>>>>>>>> egg_info: >>>>>>>>>>>>> 10 ERROR: Traceback (most recent call last): >>>>>>>>>>>>> 11 File "<string>", line 1, in <module> >>>>>>>>>>>>> 12 File "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", >>>>>>>>>>>>> line 41, in <module> >>>>>>>>>>>>> 13 import pycodestyle >>>>>>>>>>>>> 14 ImportError: No module named 'pycodestyle' >>>>>>>>>>>>> 15 ---------------------------------------- >>>>>>>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with error >>>>>>>>>>>>> code 1 in /tmp/pip-install-wrqytf9a/avro-python3/ >>>>>>>>>>>>> >>>>>>>>>>>>
