[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt
[ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572766#comment-16572766 ] cclauss commented on BEAM-3106: --- What about moving to https://docs.pipenv.org ? > Consider not pinning all python dependencies, or moving them to > requirements.txt > > > Key: BEAM-3106 > URL: https://issues.apache.org/jira/browse/BEAM-3106 > Project: Beam > Issue Type: Wish > Components: build-system >Affects Versions: 2.1.0 > Environment: python >Reporter: Maximilian Roos >Priority: Major > > Currently all python dependencies are [pinned or > capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97] > While there's a good argument for supplying a `requirements.txt` with well > tested dependencies, having them specified in `setup.py` forces them to an > exact state on each install of Beam. This makes using Beam in any environment > with other libraries nigh on impossible. > This is particularly severe for the `gcp` dependencies, where we have > libraries that won't work with an older version (but Beam _does_ work with an > newer version). We have to do a bunch of gymnastics to get the correct > versions installed because of this. Unfortunately, airflow repeats this > practice and conflicts on a number of dependencies, adding further > complication (but, again there is no real conflict). > I haven't seen this practice outside of the Apache & Google ecosystem - for > example no libraries in numerical python do this. Here's a [discussion on > SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-1251) Python 3 Support
[ https://issues.apache.org/jira/browse/BEAM-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569757#comment-16569757 ] cclauss edited comment on BEAM-1251 at 8/6/18 5:39 AM: --- Can we please upgrade *+Priority: Trivial+* for this issue and create a concise list of the tasks remaining to close it? ~500 days until Python 2 end of life. was (Author: cclauss): Can we please upgrade *+Priority: Trivial+* for this issue can create a concise list of the tasks remaining to close it? ~500 days until Python 2 end of life. > Python 3 Support > > > Key: BEAM-1251 > URL: https://issues.apache.org/jira/browse/BEAM-1251 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Eyad Sibai >Assignee: Robbe >Priority: Trivial > Time Spent: 18.5h > Remaining Estimate: 0h > > I have been trying to use google datalab with python3. As I see there are > several packages that does not support python3 yet which google datalab > depends on. This is one of them. > https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-1251) Python 3 Support
[ https://issues.apache.org/jira/browse/BEAM-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569757#comment-16569757 ] cclauss commented on BEAM-1251: --- Can we please upgrade *+Priority: Trivial+* for this issue can create a concise list of the tasks remaining to close it? ~500 days until Python 2 end of life. > Python 3 Support > > > Key: BEAM-1251 > URL: https://issues.apache.org/jira/browse/BEAM-1251 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Eyad Sibai >Assignee: Robbe >Priority: Trivial > Time Spent: 18.5h > Remaining Estimate: 0h > > I have been trying to use google datalab with python3. As I see there are > several packages that does not support python3 yet which google datalab > depends on. This is one of them. > https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3959) Eliminating Python "undefined names"
cclauss created BEAM-3959: - Summary: Eliminating Python "undefined names" Key: BEAM-3959 URL: https://issues.apache.org/jira/browse/BEAM-3959 Project: Beam Issue Type: Bug Components: sdk-py-core, sdk-py-harness Reporter: cclauss Assignee: Ahmet Altay My sense is that E901,E999,F821,F822,F823 are the "showstopper" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. The other flake8 issues are merely "style violations" -- useful for readability but they do not effect runtime safety. Work done under https://issues.apache.org/jira/browse/BEAM-3738 has enabled Jenkins to run linters on Python 3 code. One of the advantages of the *flake8* linter is that it can detect "undefined names" in Python code so it has been added to the [Python 2|https://github.com/apache/beam/tree/master/sdks/python/run_pylint.sh] and [Python 3|https://github.com/apache/beam/tree/master/sdks/python/run_mini_py3lint.sh] test runs. https://github.com/apache/beam/pull/4798 currently lists 23 undefined names when flake8 is run under Python 3. Solutions to some of these undefined names can be found in: * https://github.com/apache/beam/pull/4561 * https://github.com/apache/beam/pull/4774 * https://github.com/apache/beam/pull/4820 Once the Beam codebase can pass a flake8 F821 test with no undefined names, then F821 should be added to the the other "showstopper" issues as described in https://github.com/apache/beam/pull/4798 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3872) BaseException.message deprecated as of Python 2.6 and removed in Python 3
[ https://issues.apache.org/jira/browse/BEAM-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cclauss updated BEAM-3872: -- Description: >From the Travis logs... ./sdks/python/apache_beam/typehints/typecheck.py:179: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 six.raise_from(TypeCheckError(e.message), sys.exc_info()[2]) e.args[0] should be used in place of e.message was: >From the Travis logs... ./sdks/python/apache_beam/typehints/typecheck.py:179: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 six.raise_from(TypeCheckError(e.message), sys.exc_info()[2]) e.args[1] should be used in place of e.message > BaseException.message deprecated as of Python 2.6 and removed in Python 3 > - > > Key: BEAM-3872 > URL: https://issues.apache.org/jira/browse/BEAM-3872 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: cclauss >Assignee: Ahmet Altay >Priority: Minor > Original Estimate: 1m > Remaining Estimate: 1m > > From the Travis logs... > ./sdks/python/apache_beam/typehints/typecheck.py:179: DeprecationWarning: > BaseException.message has been deprecated as of Python 2.6 > six.raise_from(TypeCheckError(e.message), sys.exc_info()[2]) > e.args[0] should be used in place of e.message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3872) BaseException.message deprecated as of Python 2.6 and removed in Python 3
cclauss created BEAM-3872: - Summary: BaseException.message deprecated as of Python 2.6 and removed in Python 3 Key: BEAM-3872 URL: https://issues.apache.org/jira/browse/BEAM-3872 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: cclauss Assignee: Ahmet Altay >From the Travis logs... ./sdks/python/apache_beam/typehints/typecheck.py:179: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 six.raise_from(TypeCheckError(e.message), sys.exc_info()[2]) e.args[1] should be used in place of e.message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3671) Need Python 3.x interpreter in Jenkins machines
[ https://issues.apache.org/jira/browse/BEAM-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386291#comment-16386291 ] cclauss commented on BEAM-3671: --- I also believe that this issue can be closed. > Need Python 3.x interpreter in Jenkins machines > --- > > Key: BEAM-3671 > URL: https://issues.apache.org/jira/browse/BEAM-3671 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Ahmet Altay >Assignee: Alan Myrvold >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > This is required for being able to test with Py3. If possible, it would be > best to have 3.5 and 3.6. If not we can settle on a single version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3769) Jira link is broken
cclauss created BEAM-3769: - Summary: Jira link is broken Key: BEAM-3769 URL: https://issues.apache.org/jira/browse/BEAM-3769 Project: Beam Issue Type: Bug Components: website Reporter: cclauss Assignee: Reuven Lax [https://github.com/apache/beam#sdks] click the __JIRA__ link... It is a broken link -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3670) DirectRunner: TypeError: 'KeyedWorkItem' object is not iterable
[ https://issues.apache.org/jira/browse/BEAM-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358781#comment-16358781 ] cclauss commented on BEAM-3670: --- The last message talks about **line 1453** but the text it is showing is from the next function on **line 1456**. [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py#L1453-L1456] Are we sure that it is using to code that is currently on **master**? Also Python 2.7.5 is raising a bunch of requests/urllib3 warnings that are fixed in 2.7.9+... Current Python 2 is 2.7.14. > DirectRunner: TypeError: 'KeyedWorkItem' object is not iterable > --- > > Key: BEAM-3670 > URL: https://issues.apache.org/jira/browse/BEAM-3670 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Assignee: Charles Chen >Priority: Major > > I see the following error in a pre-commit test with no code changes: > [https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/2661/org.apache.beam$beam-sdks-python/console] > > root: ERROR: Exception at bundle > , > due to an exception. > Traceback (most recent call last): > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/executor.py", > line 305, in call > side_input_values) > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/executor.py", > line 338, in attempt_call > evaluator.process_element(value) > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/runners/direct/transform_evaluator.py", > line 560, in process_element > self.runner.process(element) > File "apache_beam/runners/common.py", line 468, in > apache_beam.runners.common.DoFnRunner.process > (apache_beam/runners/common.c:13915) > def process(self, windowed_value): > File "apache_beam/runners/common.py", line 474, in > apache_beam.runners.common.DoFnRunner.process > (apache_beam/runners/common.c:13719) > self._reraise_augmented(exn) > File "apache_beam/runners/common.py", line 515, in > apache_beam.runners.common.DoFnRunner._reraise_augmented > (apache_beam/runners/common.c:14934) > raise new_exn, None, original_traceback > File "apache_beam/runners/common.py", line 472, in > apache_beam.runners.common.DoFnRunner.process > (apache_beam/runners/common.c:13632) > self.do_fn_invoker.invoke_process(windowed_value) > File "apache_beam/runners/common.py", line 270, in > apache_beam.runners.common.SimpleInvoker.invoke_process > (apache_beam/runners/common.c:9084) > windowed_value, self.process_method(windowed_value.value)) > File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/src/sdks/python/apache_beam/transforms/core.py", > line 1453, in process > k, vs = element > TypeError: 'KeyedWorkItem' object is not iterable [while running > 'assert_that/Group/GroupByKey/GroupByWindow'] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3591) Undefined name: exc_info
cclauss created BEAM-3591: - Summary: Undefined name: exc_info Key: BEAM-3591 URL: https://issues.apache.org/jira/browse/BEAM-3591 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: cclauss Assignee: Ahmet Altay __exc_info__ is an undefined name which might result in NameError being raised instead of the desired error. Proposed fix is https://github.com/apache/beam/pull/4559 flake8 testing of https://github.com/apache/beam $ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__ ``` ./sdks/python/apache_beam/runners/worker/data_plane.py:185:19: F821 undefined name 'exc_info' raise exc_info[0], exc_info[1], exc_info[2] ^ ./sdks/python/apache_beam/runners/worker/data_plane.py:185:32: F821 undefined name 'exc_info' raise exc_info[0], exc_info[1], exc_info[2] ^ ./sdks/python/apache_beam/runners/worker/data_plane.py:185:45: F821 undefined name 'exc_info' raise exc_info[0], exc_info[1], exc_info[2] ^ ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
