[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17131125#comment-17131125
]
Beam JIRA Bot commented on BEAM-8966:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Priority: P2
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121755#comment-17121755
]
Kenneth Knowles commented on BEAM-8966:
---
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: P2
> Labels: stale-assigned
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999019#comment-16999019
]
Kamil Wasilewski commented on BEAM-8966:
Fixed by https://github.com/apache/beam/pull/10400
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998078#comment-16998078
]
Kamil Wasilewski commented on BEAM-8966:
I've filled https://issues.apache.org/jira/browse/BEAM-8979 to track the issue
I described.
> In the meantime, the mypy tests are not required – they don't actually pass
> yet, they're simply there for reference – so we could remove the
> mypy-protobuf change and disable the tests until after the pep517 changes.
We could at least disable mypy stub files generating[1] for now, so that the
protoc-gen-mypy won't be executed.
[1]
https://github.com/apache/beam/blob/4875f4576f6d17ba331ab386d583d99ab4ce1a2d/sdks/python/gen_protos.py#L136
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997522#comment-16997522
]
Udi Meiri commented on BEAM-8966:
-
There's no reason why ITs don't go through tox, and I believe they should.
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997506#comment-16997506
]
Chad Dombrova commented on BEAM-8966:
-
Ok, I'm working on trying to reproduce this, but running into errors. This
test copies the entirety of the sdks/python directory, including all of my
intermediate build, target, and tox env directories, and that's tripping
something up. Seems like another example of a highly inefficient approach to
creating build isolation. Is there a reason the ITs don't go through tox?
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997486#comment-16997486
]
Udi Meiri commented on BEAM-8966:
-
I get the same "ModuleNotFoundError" error if I run this on my local machine:
{code}
./gradlew :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
{code}
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997438#comment-16997438
]
Chad Dombrova commented on BEAM-8966:
-
hmmm... the errors show two different problems.
In Udi's, mypy-protobuf is newly installed, protoc-gen-mypy is found by
protoc, but there's a failure to import google.protobuf. In Kamil's
mypy-protobuf is already installed, but protoc-gen-mypy is not found by protoc.
Does anyone have a way to reproduce this reliably? What about locally?
I have a hunch this is the kind of problem that would be solved by the pep517
efforts I've been working on. In the meantime, the mypy tests are not required
– they don't actually pass yet, they're simply there for reference – so we
could remove the mypy-protobuf change and disable the tests until after the
pep517 changes.
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
> _bootstrap
> self.run()
> File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
> self._target(*self._args, **self._kwargs)
> File "/app/sdks/python/gen_protos.py", line 189, in
> _install_grpcio_tools_and_generate_proto_files
> generate_proto_files()
> File "/app/sdks/python/gen_protos.py", line 144, in generate_proto_files
> '%s' % ret_code)
> RuntimeError: Protoc returned non-zero status (see logs for details): 1
> [0m[91mTraceback (most recent call last):
> File "/app/sdks/python/gen_protos.py", line 104, in generate_proto_files
> from grpc_tools import protoc
> ModuleNotFoundError: No module named 'grpc_tools'
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "setup.py", line 295, in
> 'mypy': generate_protos_first(mypy),
> File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
> 145, in setup
> return distutils.core.setup(**attrs)
> File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
> self.run_command(cmd)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "/usr/local/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> self.run_command('egg_info')
> File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
> self.distribution.run_command(command)
> File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
> cmd_obj.run()
> File "setup.py", line 220, in run
> gen_protos.generate_proto_files(log=log)
> File "/app/sdks/python/gen_protos.py", line 121, in generate_proto_files
> raise ValueError("Proto generation failed (see log for details).")
> ValueError: Proto generation failed (see log for details).
> [0mService 'test' failed to build: The command '/bin/sh -c cd sdks/python &&
> python setup.py sdist && pip install --no-cache-dir $(ls
> dist/apache-beam-*.tar.gz | tail -n1)[gcp]' returned a non-zero code: 1
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1114/consoleText
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8966) failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
[
https://issues.apache.org/jira/browse/BEAM-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997429#comment-16997429
]
Kamil Wasilewski commented on BEAM-8966:
Similar error happens in
[https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
|https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]and
[https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
No idea why not every Python tests are affected. Also, I'm not able to
reproduce this error locally.
{code:java}
16:52:40 > Task :sdks:python:sdist FAILED
16:52:40 setup.py:232: UserWarning: You are using Apache Beam with Python 2.
New releases of Apache Beam will soon support Python 3 only.
16:52:40 'You are using Apache Beam with Python 2. '
16:52:40
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/src/build/gradleenv/192237/local/lib/python2.7/site-packages/setuptools/dist.py:476:
UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
16:52:40 normalized_version,
16:52:40 DEPRECATION: Python 2.7 will reach the end of its life on January 1st,
2020. Please upgrade your Python as Python 2.7 won't be maintained after that
date. A future version of pip will drop support for Python 2.7. More details
about Python 2 support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
16:52:40 Requirement already satisfied: mypy-protobuf==1.12 in
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/src/build/gradleenv/192237/lib/python2.7/site-packages
(1.12)
16:52:40 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
but not used.
16:52:40 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but
not used.
16:52:40 protoc-gen-mypy: program not found or is not executable
16:52:40 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
16:52:40 Traceback (most recent call last):
16:52:40 File "setup.py", line 295, in
16:52:40 'mypy': generate_protos_first(mypy),
16:52:40 File
"/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/src/build/gradleenv/192237/local/lib/python2.7/site-packages/setuptools/__init__.py",
line 145, in setup
16:52:40 return distutils.core.setup(**attrs)
16:52:40 File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
16:52:40 dist.run_commands()
16:52:40 File "/usr/lib/python2.7/distutils/dist.py", line 953, in
run_commands
16:52:40 self.run_command(cmd)
16:52:40 File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
16:52:40 cmd_obj.run()
16:52:40 File
"/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/src/build/gradleenv/192237/local/lib/python2.7/site-packages/setuptools/command/sdist.py",
line 44, in run
16:52:40 self.run_command('egg_info')
16:52:40 File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
16:52:40 self.distribution.run_command(command)
16:52:40 File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
16:52:40 cmd_obj.run()
16:52:40 File "setup.py", line 220, in run
16:52:40 gen_protos.generate_proto_files(log=log)
16:52:40 File
"/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/src/sdks/python/gen_protos.py",
line 144, in generate_proto_files
16:52:40 '%s' % ret_code)
16:52:40 RuntimeError: Protoc returned non-zero status (see logs for details): 1
{code}
> failure in :sdks:python:test-suites:direct:py37:hdfsIntegrationTest
> ---
>
> Key: BEAM-8966
> URL: https://issues.apache.org/jira/browse/BEAM-8966
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Chad Dombrova
>Priority: Major
>
> I believe this is due to https://github.com/apache/beam/pull/9915
> {code}
> [0mCollecting mypy-protobuf==1.12
> Using cached
> https://files.pythonhosted.org/packages/b6/28/041dea47c93564bfc0ece050362894292ec4f173caa92fa82994a6d061d1/mypy_protobuf-1.12-py3-none-any.whl
> Installing collected packages: mypy-protobuf
> Successfully installed mypy-protobuf-1.12
> [91mbeam_fn_api.proto: warning: Import google/protobuf/descriptor.proto but
> not used.
> beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto but not
> used.
> [0m[91mTraceback (most recent call last):
> File "/usr/local/bin/protoc-gen-mypy", line 13, in
> import google.protobuf.descriptor_pb2 as d
> ModuleNotFoundError: No module named 'google'
> [0m[91m--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> [0m[91mProcess Process-1:
> Traceback (most recent
