Thank you, Ismaël. Good to know Avro doesn't follow semantic versioning.
Replied on the PR.

On Thu, Feb 13, 2020 at 5:24 AM Ismaël Mejía <ieme...@gmail.com> wrote:

> For info Avro has published a new version 1.9.2.1 that fixes the issue:
> https://issues.apache.org/jira/browse/AVRO-2737
>
> I just submitted a PR to make the dependency consistent with Avro
> versioning and
> verify that everything works as intended with the upgraded dependency on
> the
> python SDK. Can you PTAL?
> https://github.com/apache/beam/pull/10851
>
>
> On Thu, Feb 13, 2020 at 9:39 AM Ismaël Mejía <ieme...@gmail.com> wrote:
>
>>
>> > I can argue for not pinning and bounding with major version ranges.
>> This gives flexibility to users to mix other third party libraries that
>> share common dependencies with Beam. Our expectation is that dependencies
>> follow semantic versioning and do not introduce breaking changes unless
>> there is a major version change. A good example of this is Beam's
>> dependency on "pytz>=2018.3". It is a simple wrapper around a time zone
>> file. Latest version of the dependency is 2019.3, it is updated a few times
>> a year. Beam users do not have to update Beam just to be able to use a
>> later version of it since Beam does not pin it.
>>
>> Avro does not follow semantic versioning (the first number corresponds to
>> the version of the Avro binary format the release is compatible with, the
>> second correspond to the MAJOR and the third to the MINOR in semver), so we
>> should then fix the upper bound to 1.10.0 instead of 2.0.0 considering that
>> 1.10.x before the summer and it may contain breaking changes.
>>
>> > There is also a middle ground, where we can pin certain dependencies if
>> we are not confident about their releases. And allow ranges for rest of the
>> dependencies. In general, we are currently following this practice.
>>
>> I see your point, like many things in software it is all about tradeoffs,
>> and it is good to find a middle ground, do we have a robust reproducible
>> release experience, or do we deal with the annoyance of doing manual minor
>> version upgrades. Choices choices...
>>
>>
>>
>>
>> On Thu, Feb 13, 2020 at 2:26 AM Ahmet Altay <al...@google.com> wrote:
>>
>>>
>>>
>>> On Wed, Feb 12, 2020 at 12:54 PM Ismaël Mejía <ieme...@gmail.com> wrote:
>>>
>>>> Independently of the bug in the dependency release the fact that the
>>>> Beam Python
>>>> SDK does not have pinned fixed dependency numbers is error-prone. We may
>>>> continue to have this kind of problems until we fix this (with other
>>>> dependencies too). In the Java SDK we do not accept such type of dynamic
>>>> dependency numbers and python should probably follow this practice to
>>>> avoid
>>>> issues like the present one.
>>>>
>>>> Why don't we just do:
>>>>
>>>>     'avro-python3==1.9.1',
>>>>
>>>> instead of the current:
>>>>
>>>>     'avro-python3>=1.8.1,!=1.9.2,<2.0.0; python_version >= "3.0"',
>>>>
>>>
>>> I agree this is error prone. Your argument for pinning makes sense and I
>>> agree with it.
>>>
>>> I can argue for not pinning and bounding with major version ranges. This
>>> gives flexibility to users to mix other third party libraries that share
>>> common dependencies with Beam. Our expectation is that dependencies follow
>>> semantic versioning and do not introduce breaking changes unless there is a
>>> major version change. A good example of this is Beam's dependency on
>>> "pytz>=2018.3". It is a simple wrapper around a time zone file. Latest
>>> version of the dependency is 2019.3, it is updated a few times a year. Beam
>>> users do not have to update Beam just to be able to use a later version of
>>> it since Beam does not pin it.
>>>
>>> There is also a middle ground, where we can pin certain dependencies if
>>> we are not confident about their releases. And allow ranges for rest of the
>>> dependencies. In general, we are currently following this practice.
>>>
>>>
>>>>
>>>>
>>>> On Wed, Feb 12, 2020 at 9:14 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Related: we have dependencies on avro, avro-python3, and fastavro.
>>>>> fastavro supports both python 2 and 3. Could we reduce this dependency 
>>>>> list
>>>>> and depend only on fastavro? If we need avro and avro-python3 for the
>>>>> purposes of testing only, we can move them to test only dependencies.
>>>>>
>>>>> +Chamikara Jayalath <chamik...@google.com>, because I vaguely
>>>>> remember him working on this.
>>>>>
>>>>> The reason I am calling for this is the impact of bad dependency
>>>>> releases are high. All previously released Beam versions will be impacted.
>>>>> Reducing the dependency list will reduce the risk.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> On Wed, Feb 12, 2020 at 12:02 PM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>> Thank you Valentyn!
>>>>>>
>>>>>> On Wed, Feb 12, 2020 at 11:32 AM Valentyn Tymofieiev <
>>>>>> valen...@google.com> wrote:
>>>>>>
>>>>>>> Yes, otherwise all Python tests will continue to fail until Avro
>>>>>>> comes up with a new release. Sent:
>>>>>>> https://github.com/apache/beam/pull/10844
>>>>>>>
>>>>>>> On Wed, Feb 12, 2020 at 11:08 AM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Should we update Beam's setup.py to skip this avro-python3 version?
>>>>>>>>
>>>>>>>> On Wed, Feb 12, 2020 at 10:57 AM Alan Krumholz <
>>>>>>>> alan.krumh...@betterup.co> wrote:
>>>>>>>>
>>>>>>>>> makes sense. I'll add this workaround for now.
>>>>>>>>> Thanks so much for your help!
>>>>>>>>>
>>>>>>>>> On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev <
>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Alan, Dataflow workers preinstall Beam SDK dependencies,
>>>>>>>>>> including (a working version) of avro-python3. So after reading your 
>>>>>>>>>> email
>>>>>>>>>> once again, I think in your case you were not able to install Beam 
>>>>>>>>>> SDK
>>>>>>>>>> locally. So a workaround for you would be to `pip install
>>>>>>>>>> avro-python3==1.9.1` or `pip install pycodestyle`  before installing 
>>>>>>>>>> Beam,
>>>>>>>>>> until AVRO-2737 is resolved.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 12, 2020 at 10:21 AM Valentyn Tymofieiev <
>>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ah, there's already
>>>>>>>>>>> https://issues.apache.org/jira/browse/AVRO-2737 and it received
>>>>>>>>>>> attention.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev <
>>>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Opened https://issues.apache.org/jira/browse/AVRO-2738
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev <
>>>>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Here's a short repro:
>>>>>>>>>>>>>
>>>>>>>>>>>>> :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch
>>>>>>>>>>>>> root@04b45a100d16:/# pip install avro-python3
>>>>>>>>>>>>> Collecting avro-python3
>>>>>>>>>>>>>   Downloading avro-python3-1.9.2.tar.gz (37 kB)
>>>>>>>>>>>>>     ERROR: Command errored out with exit status 1:
>>>>>>>>>>>>>      command: /usr/local/bin/python -c 'import sys,
>>>>>>>>>>>>> setuptools, tokenize; sys.argv[0] =
>>>>>>>>>>>>> '"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';
>>>>>>>>>>>>> __file__='"'"'/tmp/pip-install-mmy4vspt/avro-python3/setup.py'"'"';f=getattr(tokenize,
>>>>>>>>>>>>> '"'"'open'"'"', 
>>>>>>>>>>>>> open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
>>>>>>>>>>>>> '"'"'\n'"'"');f.close();exec(compile(code, __file__, 
>>>>>>>>>>>>> '"'"'exec'"'"'))'
>>>>>>>>>>>>> egg_info --egg-base 
>>>>>>>>>>>>> /tmp/pip-install-mmy4vspt/avro-python3/pip-egg-info
>>>>>>>>>>>>>          cwd: /tmp/pip-install-mmy4vspt/avro-python3/
>>>>>>>>>>>>>     Complete output (5 lines):
>>>>>>>>>>>>>     Traceback (most recent call last):
>>>>>>>>>>>>>       File "<string>", line 1, in <module>
>>>>>>>>>>>>>       File "/tmp/pip-install-mmy4vspt/avro-python3/setup.py",
>>>>>>>>>>>>> line 41, in <module>
>>>>>>>>>>>>>         import pycodestyle
>>>>>>>>>>>>>     ModuleNotFoundError: No module named 'pycodestyle'
>>>>>>>>>>>>>     ----------------------------------------
>>>>>>>>>>>>> ERROR: Command errored out with exit status 1: python setup.py
>>>>>>>>>>>>> egg_info Check the logs for full command output.
>>>>>>>>>>>>> root@04b45a100d16:/#
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev <
>>>>>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, it is a bug in the recent Avro release. We should report
>>>>>>>>>>>>>> it to the Avro maintainers. The workaround is to downgrade 
>>>>>>>>>>>>>> avro-python3 to
>>>>>>>>>>>>>> 1.9.1, for example via requirements.txt.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz <
>>>>>>>>>>>>>> sniem...@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> avro-python3 1.9.2 was released on pypi 4 hours ago, and
>>>>>>>>>>>>>>> added pycodestyle as a dependency, probably related?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik <lc...@google.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +dev <dev@beam.apache.org>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There was recently an update to add autoformatting to the
>>>>>>>>>>>>>>>> Python SDK[1].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm seeing this during testing of a PR as well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1:
>>>>>>>>>>>>>>>> https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Feb 12, 2020 at 9:57 AM Alan Krumholz <
>>>>>>>>>>>>>>>> alan.krumh...@betterup.co> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Some more information for this as I still can't get to fix
>>>>>>>>>>>>>>>>> it....
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This job is triggered using the beam[gcp] python sdk from
>>>>>>>>>>>>>>>>> a KubeFlow Pipelines component which runs on top of docker 
>>>>>>>>>>>>>>>>> image:
>>>>>>>>>>>>>>>>> tensorflow/tensorflow:1.13.1-py3
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I just checked and that image hasn't been updated
>>>>>>>>>>>>>>>>> recently. I also redeployed my pipeline to another (older) 
>>>>>>>>>>>>>>>>> deployment of
>>>>>>>>>>>>>>>>> KFP and it gives me the same error (which tells me this isn't 
>>>>>>>>>>>>>>>>> an internal
>>>>>>>>>>>>>>>>> KFP problem)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The exact same pipeline/code running on the exact same
>>>>>>>>>>>>>>>>> image has been running fine for days. Did anything changed on 
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> beam/dataflow side since yesterday morning?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for your help! this is a production pipeline that
>>>>>>>>>>>>>>>>> is not running for us :(
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Feb 12, 2020 at 7:21 AM Alan Krumholz <
>>>>>>>>>>>>>>>>> alan.krumh...@betterup.co> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi, I have a scheduled daily job that I have been running
>>>>>>>>>>>>>>>>>> fine in dataflow for days now.
>>>>>>>>>>>>>>>>>> We haven't changed anything on this code but this morning
>>>>>>>>>>>>>>>>>> run failed  (it couldn't even spin up the job)
>>>>>>>>>>>>>>>>>> The job submits a setup.py file (that also hasn't
>>>>>>>>>>>>>>>>>> changed) but maybe is causing the problem? (based on the 
>>>>>>>>>>>>>>>>>> error I'm getting)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Anyone else having the same issue? or know how to fix it?
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ERROR: Complete output from command python setup.py
>>>>>>>>>>>>>>>>>> egg_info:
>>>>>>>>>>>>>>>>>> 2 ERROR: Traceback (most recent call last):
>>>>>>>>>>>>>>>>>> 3 File "<string>", line 1, in <module>
>>>>>>>>>>>>>>>>>> 4 File
>>>>>>>>>>>>>>>>>> "/tmp/pip-install-42zyi89t/avro-python3/setup.py", line 41, 
>>>>>>>>>>>>>>>>>> in <module>
>>>>>>>>>>>>>>>>>> 5 import pycodestyle
>>>>>>>>>>>>>>>>>> 6 ImportError: No module named 'pycodestyle'
>>>>>>>>>>>>>>>>>> 7 ----------------------------------------
>>>>>>>>>>>>>>>>>> 8ERROR: Command "python setup.py egg_info" failed with
>>>>>>>>>>>>>>>>>> error code 1 in /tmp/pip-install-42zyi89t/avro-python3/
>>>>>>>>>>>>>>>>>> 9 ERROR: Complete output from command python setup.py
>>>>>>>>>>>>>>>>>> egg_info:
>>>>>>>>>>>>>>>>>> 10 ERROR: Traceback (most recent call last):
>>>>>>>>>>>>>>>>>> 11 File "<string>", line 1, in <module>
>>>>>>>>>>>>>>>>>> 12 File
>>>>>>>>>>>>>>>>>> "/tmp/pip-install-wrqytf9a/avro-python3/setup.py", line 41, 
>>>>>>>>>>>>>>>>>> in <module>
>>>>>>>>>>>>>>>>>> 13 import pycodestyle
>>>>>>>>>>>>>>>>>> 14 ImportError: No module named 'pycodestyle'
>>>>>>>>>>>>>>>>>> 15 ----------------------------------------
>>>>>>>>>>>>>>>>>> 16ERROR: Command "python setup.py egg_info" failed with
>>>>>>>>>>>>>>>>>> error code 1 in /tmp/pip-install-wrqytf9a/avro-python3/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>

Reply via email to