[
https://issues.apache.org/jira/browse/BEAM-8979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076407#comment-17076407
]
yoshiki obata commented on BEAM-8979:
-------------------------------------
I found this issue reproduced on following condition.
- on Python 3.7.6 docker container
- no packages installed by pip
- checkout master(5820ab6) and run command `python setup.py install` at
sdks/python
error log
{code:java}
running install
running bdist_egg
running egg_info
/usr/local/lib/python3.7/site-packages/setuptools/dist.py:476: UserWarning:
Normalizing '2.21.0.dev' to '2.21.0.dev0'
normalized_version,
/home/beam/sdks/python/gen_protos.py:241: UserWarning: Installing grpcio-tools
is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
WARNING:root:Installing grpcio-tools into
/home/beam/sdks/python/.eggs/grpcio-wheels
Collecting grpcio-tools<=1.14.2,>=1.3.5
Downloading
https://files.pythonhosted.org/packages/c1/7d/048c311755118cf574a8b072633586ef113dfc5993c841981d2371de7c5d/grpcio_tools-1.14.2-cp37-cp37m-manylinux1_x86_64.whl
(22.2MB)
Collecting future==0.16.0
Downloading
https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz
(824kB)
Collecting mypy-protobuf==1.18
Downloading
https://files.pythonhosted.org/packages/7f/71/33576c19ce209b18a21cb494c7f7930fd73135d95d4e0c53b4ad866390ee/mypy_protobuf-1.18-py3-none-any.whl
Collecting protobuf>=3.5.0.post1
Downloading
https://files.pythonhosted.org/packages/ff/f1/8dcd4219bbae8aa44fe8871a89f05eca2dca9c04f8dbfed8a82b7be97a88/protobuf-3.11.3-cp37-cp37m-manylinux1_x86_64.whl
(1.3MB)
Collecting grpcio>=1.14.2
Downloading
https://files.pythonhosted.org/packages/f1/f4/c437929e1a8db457d85c0c16d348a5dd00427114e281a9a0579c07114cf6/grpcio-1.28.1-cp37-cp37m-manylinux2010_x86_64.whl
(2.8MB)
Collecting setuptools
Downloading
https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl
(582kB)
Collecting six>=1.9
Downloading
https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Building wheels for collected packages: future
Building wheel for future (setup.py): started
Building wheel for future (setup.py): finished with status 'done'
Created wheel for future: filename=future-0.16.0-cp37-none-any.whl
size=487797
sha256=a0e21609ddf7ee8725b901d171ea12c873c024e470794f8bcf0de35e1cdf69a7
Stored in directory:
/root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a
Successfully built future
Installing collected packages: setuptools, six, protobuf, grpcio, grpcio-tools,
future, mypy-protobuf
Successfully installed future-0.16.0 grpcio-1.28.1 grpcio-tools-1.14.2
mypy-protobuf-1.18 protobuf-3.11.3 setuptools-46.1.3 six-1.14.0
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
WARNING:root:Installing grpcio-tools took 6.37 seconds.
INFO:gen_protos:Regenerating Python proto definitions (no output files).
Process Process-1:
Traceback (most recent call last):
File "/home/beam/sdks/python/gen_protos.py", line 292, in generate_proto_files
from grpc_tools import protoc
ModuleNotFoundError: No module named 'grpc_tools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in
_bootstrap
self.run()
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/beam/sdks/python/gen_protos.py", line 378, in
_install_grpcio_tools_and_generate_proto_files
generate_proto_files(force=force)
File "/home/beam/sdks/python/gen_protos.py", line 315, in generate_proto_files
protoc_gen_mypy = _find_protoc_gen_mypy()
File "/home/beam/sdks/python/gen_protos.py", line 233, in
_find_protoc_gen_mypy
(fname, ', '.join(search_paths)))
RuntimeError: Could not find protoc-gen-mypy in /usr/local/bin, /usr/local/bin,
/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin
Traceback (most recent call last):
File "/home/beam/sdks/python/gen_protos.py", line 292, in generate_proto_files
from grpc_tools import protoc
ModuleNotFoundError: No module named 'grpc_tools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 315, in <module>
'mypy': generate_protos_first(mypy),
File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line
145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.7/site-packages/setuptools/command/install.py",
line 67, in run
self.do_egg_install()
File "/usr/local/lib/python3.7/site-packages/setuptools/command/install.py",
line 109, in do_egg_install
self.run_command('bdist_egg')
File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File
"/usr/local/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line
163, in run
self.run_command("egg_info")
File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "setup.py", line 239, in run
gen_protos.generate_proto_files()
File "/home/beam/sdks/python/gen_protos.py", line 310, in generate_proto_files
raise ValueError("Proto generation failed (see log for details).")
ValueError: Proto generation failed (see log for details).
{code}
> protoc-gen-mypy: program not found or is not executable
> -------------------------------------------------------
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
> Reporter: Kamil Wasilewski
> Assignee: Chad Dombrova
> Priority: Major
> Fix For: Not applicable
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in <module>
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
