[
https://issues.apache.org/jira/browse/BEAM-1790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15938947#comment-15938947
]
Ahmet Altay commented on BEAM-1790:
-----------------------------------
[~mlambert] Where do you get this error, is this a at job creation time or
execution time?
I managed to reproduce this error with and old (unsupported by Beam) verison of
pip (1.5.4). Could you check your installed pip version?
> Failure to build --requirements.txt when it uses google protobuf
> ----------------------------------------------------------------
>
> Key: BEAM-1790
> URL: https://issues.apache.org/jira/browse/BEAM-1790
> Project: Beam
> Issue Type: Bug
> Components: sdk-py
> Reporter: Mike Lambert
> Assignee: Ahmet Altay
> Labels: build, requirements
>
> I am running with {{--requirements_file requirements.txt}}, which contains:
> {noformat}
> google-cloud-datastore
> {noformat}
> Unfortunately, when attempting to run this on the cloud dataflow, I get the
> following error trying to build the requirements:
> {noformat}
> Collecting setuptools (from
> protobuf>=3.0.0->google-cloud-core<0.24dev,>=0.23.1->google-cloud-datastore->-r
> requirements.txt (line 3))
> File was already downloaded
> /var/folders/94/wngs1jw91_n2_jjjrfljtqrc0000gn/T/dataflow-requirements-cache/setuptools-34.3.2.zip
> Complete output from command python setup.py egg_info:
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "setuptools/__init__.py", line 12, in <module>
> import setuptools.version
> File "setuptools/version.py", line 1, in <module>
> import pkg_resources
> File "pkg_resources/__init__.py", line 70, in <module>
> import packaging.version
> ImportError: No module named packaging.version
> {noformat}
> Looking online https://github.com/pypa/setuptools/issues/937 , it appears
> this is due to "pip asking setuptools to build itself (from source dist),
> which is no longer supported."
> I'm not sure what the correct fix is here...since protobuf depends on
> setuptools, and a lot of Google libraries depend on protobuf. Seems there is
> no way to whitelist protobuf/setuptools as being "provided" by the beam
> runtime (ie https://github.com/pypa/pip/issues/3090).
> I'm going to try using my own setup.py next and see if I can skirt around the
> issue, but this definitely seems like a bug with beam's requirements packager
> asking for too much?
> In the case of GCE, I compile my dependencies into a docker image that
> extends the base GCE images (and lets me use binary installs), not sure
> something like that would work here?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
