FYI I've created this issue and marked it as a blocker: https://jira.apache.org/jira/browse/BEAM-9947
Essentially, the timer encoding is broken for all non-standard key coders. The fix can be found here: https://github.com/apache/beam/pull/11658 -Max On 08.05.20 18:53, Udi Meiri wrote: > +Chad Dombrova <mailto:chad...@gmail.com> , who added _find_protoc_gen_mypy. > > I'm guessing that the code > in _install_grpcio_tools_and_generate_proto_files creates a kind of > virtualenv, but it only works well for staging Python modules and not > binaries like protoc-gen-mypy. > (I assume there's a reason why it doesn't invoke virtualenv, probably > since the list of things setup.py can expect to be installed is very > minimal (setuptools).) > > One solution would be to make these setup.py dependencies explicit in > pyproject.toml, such that pip installs them before running > setup.py: https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support > It would help when using tools like pip ("pip wheel"), but I'm not sure > what the alternative for "python setup.py sdist" is. > > > On Thu, May 7, 2020 at 10:40 PM Thomas Weise <t...@apache.org > <mailto:t...@apache.org>> wrote: > > No additional stacktraces. Full error output below. > > It's not clear what is going wrong. > > There isn't any exception from the subprocess execution since the > "WARNING:root:Installing grpcio-tools took 305.39 seconds." is printed. > > Also, the time it takes to perform the install is equivalent to > successfully running the pip command. > > I will report back if I find anything else. Currently doing the > explicit install via pip install -r sdks/python/build-requirements.txt > > Thanks, > Thomas > > WARNING:root:Installing grpcio-tools took 269.27 seconds. > INFO:gen_protos:Regenerating Python proto definitions (no output files). > Process Process-1: > Traceback (most recent call last): > File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in > _bootstrap > self.run() > File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run > self._target(*self._args, **self._kwargs) > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 378, in _install_grpcio_tools_and_generate_proto_files > generate_proto_files(force=force) > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 315, in generate_proto_files > protoc_gen_mypy = _find_protoc_gen_mypy() > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 233, in _find_protoc_gen_mypy > (fname, ', '.join(search_paths))) > RuntimeError: Could not find protoc-gen-mypy in > /code/venvs/venv2/bin, /code/venvs/venv2/bin, /code/venvs/venv3/bin, > /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin > Traceback (most recent call last): > File "setup.py", line 311, in <module> > 'mypy': generate_protos_first(mypy), > File > > "/code/venvs/venv2/local/lib/python2.7/site-packages/setuptools/__init__.py", > line 129, in setup > return distutils.core.setup(**attrs) > File "/usr/lib/python2.7/distutils/core.py", line 151, in setup > dist.run_commands() > File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands > self.run_command(cmd) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File > > "/code/venvs/venv2/local/lib/python2.7/site-packages/wheel/bdist_wheel.py", > line 204, in run > self.run_command('build') > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run > self.run_command(cmd_name) > File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "setup.py", line 235, in run > gen_protos.generate_proto_files() > File > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", line > 310, in generate_proto_files > raise ValueError("Proto generation failed (see log for details).") > ValueError: Proto generation failed (see log for details). > > > On Thu, May 7, 2020 at 2:25 PM Udi Meiri <eh...@google.com > <mailto:eh...@google.com>> wrote: > > It's hard to say without more details what's going on. Ahmet > you're right that it installs build-requirements.txt and retries > calling generate_proto_files(). > > Thomas, were there additional stacktraces? (after a "During > handling of the above exception, another exception occurred:" > message?) > > > On Thu, May 7, 2020 at 11:59 AM Ahmet Altay <al...@google.com > <mailto:al...@google.com>> wrote: > > > > On Thu, May 7, 2020 at 11:56 AM Thomas Weise <t...@apache.org > <mailto:t...@apache.org>> wrote: > > Thanks Udi! This is the issue. I'm trying to upgrade > from 2.18 where build-requirements.txt didn't exist. > > Is there a reason why this cannot happen automatically > when running python3.6 setup.py sdist bdist_wheel ? > > > I _believe_ this should happen automatically > here: > https://github.com/apache/beam/blob/master/sdks/python/gen_protos.py#L365. > Maybe there is a problem there? > > > > Thomas > > > On Thu, May 7, 2020 at 11:07 AM Udi Meiri > <eh...@google.com <mailto:eh...@google.com>> wrote: > > Probably not the issue, but double checking: are you > running "pip install -r > sdks/python/build-requirements.txt" first? > > On Wed, May 6, 2020 at 7:22 PM Thomas Weise > <t...@apache.org <mailto:t...@apache.org>> wrote: > > I'm working on rebasing our fork to 2.21.0 and > run into a problem installing grpcio-tools that > leads to *ModuleNotFoundError: No module named > 'grpc_tools' *(see details below) > > I cannot reproduce this locally. > > Any =suggestions on what to look for? > > Thanks, > Thomas > > [?25hBuilding wheels for collected packages: > future, gr > Running setup.py bdist_wheel for future ... > [?25l- \ > [?25h Stored in directory: > > /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf1 > Running setup.py bdist_wheel for grpcio ... > [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ > | / - \ | / - \ | / - \ | > [?25h Stored in directory: > > /root/.cache/pip/wheels/00/4d/5f/07d0d4283911d2b917b867a11b1622d9d2cc8c286eefd1 > Successfully built future grpcio > Installing collected packages: six, grpcio, > setuptools, protobuf, grpcio-tools, future, > mypy-protobuf > Successfully installed future-0.16.0 > grpcio-1.28.1 grpcio-tools-1.14.2 > mypy-protobuf-1.18 protobuf-3.11.3 > setuptools-46.1.3 six-1.14.0 > WARNING:root:Installing grpcio-tools took 305.39 > seconds. > INFO:gen_protos:Regenerating Python proto > definitions (no output files). > Process Process-1: > Traceback (most recent call last): > File > > "/src/streamingplatform/beam-release/beam/sdks/python/gen_protos.py", > line 292, in generate_proto_files > from grpc_tools import protoc > ModuleNotFoundError: No module named 'grpc_tools' > > On Fri, Apr 10, 2020 at 10:01 AM Kyle Weaver > <kcwea...@google.com > <mailto:kcwea...@google.com>> wrote: > > Hi everyone, > > Just a heads up that the Beam 2.21 release > branch [1] is cut. > - If you find any important issues that you > think should be addressed in the release, > please tag the jira with fix version 2.21.0 > and cc me (username `ibzib`). > - Make sure to update the change log [2] > with any significant changes if you haven't > already. Send a PR with the change and tag > me. (I imagine I'm not the only one who > forgot to do this :).) > > Thanks, > Kyle > > [1] > https://github.com/apache/beam/blob/release-2.21.0 > [2] > https://github.com/apache/beam/blob/master/CHANGES.md >
