[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036942#comment-17036942 ] Dian Fu commented on FLINK-16026: - The management of Python dependencies is quite different from the Java dependencies. Regarding to the Java dependencies, they are bundled in the distribution and so it's fine to use any version which could work. However, for the Python dependencies, they are not bundled in the distribution and we rely on the Python dependencies provided in the execution environment for Python UDF execution. The execution environment may be shared between many applications. This means that it may not be a good idea to limit the version to a very short range if possible. For example, if we limit avro-python3 to version <= 1.9.1, what will happen if users want to share the Python environment between multiple applications and another application requires a high version, e.g. 1.9.2? This introduces unnecessary dependency problems for users as we limit the version just for test stability instead because that version is not supported actually. Regarding to the avro-python3 problem, it's not caused by API breaking in the new version. It was unexpected and an error package was released just by mistaken and avro community has already provided a quick fix to address this problem. > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Assignee: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[
https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036767#comment-17036767
]
Chesnay Schepler commented on FLINK-16026:
--
{{It may be not a good idea to limit the versions of the transitive
dependencies to a different version declared by the direct dependency:}}
I disagree.
{{We should trust the direct dependency, e.g. if apache-beam declares that it
support avro-python3<2, we should trust that it does support all versions of
avro-python3 which are less than 2.}}
Not a single project can guarantee that it supports any future version of
another library, as they have no control over it.
For all we now, the next version of avro-python3 breaks the entire API.
This is just a broken concept, and we see the fallout of this repeatedly in
this project and in the JavaScript ecosystem.
{{We have to upgrade the versions of the transitive dependencies periodically
if new packages have been released}}
Correct, we have to ensure that when we bump a dependency everything still
works. I don't consider this a bad thing.
{{For released package of PyFlink, there is no way to upgrade the versions of
the transitive dependencies any more}}
This is not a problem imo, but a benefit. It means the release is stable and
will not magically stop working 2 years down the road.
> Travis failed due to python setup
> -
>
> Key: FLINK-16026
> URL: https://issues.apache.org/jira/browse/FLINK-16026
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Reporter: Jingsong Lee
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> [https://api.travis-ci.com/v3/job/286671652/log.txt]
> [https://api.travis-ci.org/v3/job/649754603/log.txt]
> [https://api.travis-ci.com/v3/job/286409130/log.txt]
> Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from
> apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached
> https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz
> Complete output from command python setup.py egg_info: Traceback (most
> recent call last): File "", line 1, in File
> "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in
> import pycodestyle ModuleNotFoundError: No module named 'pycodestyle'
> Command "python setup.py egg_info"
> failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are
> using pip version 10.0.1, however version 20.0.2 is available. You should
> consider upgrading via the 'pip install --upgrade pip' command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036656#comment-17036656 ] Dian Fu commented on FLINK-16026: - [~chesnay] Thanks a lot for your suggestions. Appreciated! We have investigated this issue and want to share our thoughts as following: - It's a good idea to limit the versions of the direct dependencies and I think we have already done that in PyFlink. - It may be not a good idea to limit the versions of the transitive dependencies to a different version declared by the direct dependency: 1. We should trust the direct dependency, e.g. if apache-beam declares that it support avro-python3<2, we should trust that it does support all versions of avro-python3 which are less than 2. 2. We should not change the range of versions of the transitive dependencies declared by the direct dependency, e.g. if apache-beam declares that it supports avro-python3<2, it may not be a good idea to change the version limit to avro-python3<1.9.1 in PyFlink. This will introduce a lot of issues: 1) We have to upgrade the versions of the transitive dependencies periodically if new packages have been released 2) For released package of PyFlink, there is no way to upgrade the versions of the transitive dependencies any more 3. If there are exceptions for the above cases, we could address them case by case. For example, regarding to the issue of this JIRA, it's because that avro community has released an error package. We could just disable it for the time being. This is a trade off, however, I guess most Python projects handle this kind of issues in this way. What's your thoughts? > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Assignee: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036088#comment-17036088 ] sunjincheng commented on FLINK-16026: - Thanks for the reminder [~chesnay]. We have discussed about initializing all current dependencies to a fixed version before this patch. The current patch is a quick fix for Travis. Will feedback here when have the final conclusion. > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Assignee: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036058#comment-17036058 ] Chesnay Schepler commented on FLINK-16026: -- Alright guys, this was now the 3rd or 4th instance of the build process being broken because some new dependency was released. Can we do a full sweep over the entire dependency tree and prevent this from happening again? [~dian.fu] [~jincheng] > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Assignee: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035921#comment-17035921 ] Dian Fu commented on FLINK-16026: - [~lzljs3620320] Thanks for reporting this issue. [~hxbks2ks] Thanks for the analysis and fix. +1 to limit the version of avro-python3 in PyFlink. > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Assignee: Huang Xingbo >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.1, 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16026) Travis failed due to python setup
[ https://issues.apache.org/jira/browse/FLINK-16026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035916#comment-17035916 ] Huang Xingbo commented on FLINK-16026: -- This issue is caused by the latest released avro package. This package is broken and it has already been discussed in avro community to release a patched version(AVRO-2737). However, I'm not sure when the new package will be available and I suggest to limit the avro-python3 version in Flink to work around this issue. I will submit a PR ASAP. > Travis failed due to python setup > - > > Key: FLINK-16026 > URL: https://issues.apache.org/jira/browse/FLINK-16026 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Jingsong Lee >Priority: Critical > Fix For: 1.11.0 > > > [https://api.travis-ci.com/v3/job/286671652/log.txt] > [https://api.travis-ci.org/v3/job/649754603/log.txt] > [https://api.travis-ci.com/v3/job/286409130/log.txt] > Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from > apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached > https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz > Complete output from command python setup.py egg_info: Traceback (most > recent call last): File "", line 1, in File > "/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in > import pycodestyle ModuleNotFoundError: No module named 'pycodestyle' > Command "python setup.py egg_info" > failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are > using pip version 10.0.1, however version 20.0.2 is available. You should > consider upgrading via the 'pip install --upgrade pip' command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
