[
https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794579#comment-16794579
]
Bruno Ripa edited comment on BEAM-3106 at 3/17/19 9:15 PM:
-----------------------------------------------------------
Actually i am still experiencing this issue, in a very simple package i am
creating for an article.
Basically in my {{setup.py}} i have two packages:
{quote}"flask==1.0.2",
"apache-beam[gcp]"
{quote}
but then i need to install {{google-cloud-storage}}; this causes, in a clean
virtual environment (Python3):
{noformat}
~/dev/dataflow/gcs_batch/df_server/dfapp » pip install -e .
j0k3r@Brunos-MacBook-Pro
Obtaining [file:///Users/j0k3r/dev/dataflow/gcs_batch/df_server/dfapp]
Collecting flask==1.0.2 (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl]
Collecting google-cloud-storage (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/b6/e9/06d9bb394fddbc62bb9c645f5e1c927128930a249d0c6a7491c3f31a9ff4/google_cloud_storage-1.14.0-py2.py3-none-any.whl]
Collecting apache-beam[gcp] (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/f7/7a/57b09ce4d6eaf7e7e192b7614a44679d6f55c11770c92f34b0ed9b59609e/apache_beam-2.11.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting itsdangerous>=0.24 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl]
Collecting Werkzeug>=0.14 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl]
Collecting click>=5.1 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl]
Collecting Jinja2>=2.10 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl]
Collecting google-api-core<2.0.0dev,>=1.6.0 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/c4/71/23a234ee35117c2ed1ebd5a62ae07ef29f9f0bae9ea816b91312bad81646/google_api_core-1.8.1-py2.py3-none-any.whl]
Collecting google-cloud-core<0.30dev,>=0.29.0 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/0c/f2/3c225e7a69cb27d283b68bff867722bd066bc1858611180197f711815ea5/google_cloud_core-0.29.1-py2.py3-none-any.whl]
Collecting google-resumable-media>=0.3.1 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/e2/5d/4bc5c28c252a62efe69ed1a1561da92bd5af8eca0cdcdf8e60354fae9b29/google_resumable_media-0.3.2-py2.py3-none-any.whl]
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting httplib2<=0.11.3,>=0.8 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting protobuf<4,>=3.5.0.post1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fb/4f/b4ab63001f81ccc3ef754965070000f056b2c13fadf93ef46ac231e3086d/protobuf-3.7.0-cp37-cp37m-macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting pyarrow<0.12.0,>=0.11.1; python_version >= "3.0" or platform_system
!= "Windows" (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/50/35/25d20077820ce8bd068bc9c06570969af7d70ef32fc453b3b41fac4c45eb/pyarrow-0.11.1-cp37-cp37m-macosx_10_6_intel.whl]
Collecting grpcio<2,>=1.8 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/df/33/c0561fe7c5e235325255f46c08bd3d07f2c80824feb22d057328eff1f8b7/grpcio-1.19.0-cp37-cp37m-macosx_10_9_x86_64.whl]
Collecting fastavro<0.22,>=0.21.4 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/78/ae/2b889d19040f11b547f550e6319e6f2602a1764e19e9c84551ece163d51e/fastavro-0.21.19-cp37-cp37m-macosx_10_13_x86_64.whl]
Collecting dill<0.2.10,>=0.2.9 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
Collecting pydot<1.3,>=1.2.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting future<1.0.0,>=0.16.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting pytz>=2018.3 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl]
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting oauth2client<4,>=2.0.1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-cloud-bigtable==0.31.1; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/00/58/8153616835b3ff7238c657400c8fc46c44b53074b39b22260dd06345f9ed/google_cloud_bigtable-0.31.1-py2.py3-none-any.whl]
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0; extra == "gcp"
(from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-apitools<0.5.27,>=0.5.26; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-cloud-pubsub==0.39.0; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fc/30/c2e6611c3ffa45816e835b016a2b40bb2bd93f05d1055f78be16a9eb2e4d/google_cloud_pubsub-0.39.0-py2.py3-none-any.whl]
Collecting google-cloud-bigquery<1.7.0,>=1.6.0; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/b7/1b/2b95f2fefddbbece38110712c225bfb5649206f4056445653bd5ca4dc86d/google_cloud_bigquery-1.6.1-py2.py3-none-any.whl]
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/ce/c6/f000f1af136ef74e4a95e33785921c73595c5390403f102e9b231b065b7a/MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl]
Collecting google-auth<2.0dev,>=0.4.0 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl]
Collecting googleapis-common-protos!=1.5.4,<2.0dev,>=1.5.3 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Collecting requests<3.0.0dev,>=2.18.0 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl]
Requirement already satisfied: six>=1.10.0 in
/Users/j0k3r/dev/virtualenvs/dfexample/lib/python3.7/site-packages (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
(1.12.0)
Requirement already satisfied: setuptools>=34.0.0 in
/Users/j0k3r/dev/virtualenvs/dfexample/lib/python3.7/site-packages (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
(40.8.0)
Collecting numpy>=1.14 (from pyarrow<0.12.0,>=0.11.1; python_version >= "3.0"
or platform_system != "Windows"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/a6/6f/cb20ccd8f0f8581e0e090775c0e3c3e335b037818416e6fa945d924397d2/numpy-1.16.2-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting pbr>=0.11 (from
mock<3.0.0,>=1.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/09/12fe9a14237a6b7e0ba3a8d6fcf254bf4b10ec56a0185f73d651145e9222/pbr-5.1.3-py2.py3-none-any.whl]
Collecting pyparsing>=2.1.4 (from
pydot<1.3,>=1.2.0->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl]
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]->dfapp==0.1.dev0)
Collecting rsa>=3.1.4 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl]
Collecting pyasn1>=0.1.7 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl]
Collecting pyasn1-modules>=0.0.5 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl]
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4 (from
google-cloud-bigtable==0.31.1; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Collecting fasteners>=0.14 (from google-apitools<0.5.27,>=0.5.26; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/3a/096c7ad18e102d4f219f5dd15951f9728ca5092a3385d2e8f79a7c1e1017/fasteners-0.14.1-py2.py3-none-any.whl]
Collecting cachetools>=2.0.0 (from
google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl]
Collecting certifi>=2017.4.17 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl]
Collecting urllib3<1.25,>=1.21.1 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl]
Collecting chardet<3.1.0,>=3.0.2 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
Collecting idna<2.9,>=2.5 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl]
Collecting monotonic>=0.1 (from
fasteners>=0.14->google-apitools<0.5.27,>=0.5.26; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl]
google-cloud-bigtable 0.31.1 has requirement
google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1
which is incompatible.
Installing collected packages: itsdangerous, Werkzeug, click, MarkupSafe,
Jinja2, flask, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, protobuf,
googleapis-common-protos, certifi, urllib3, chardet, idna, requests, pytz,
google-api-core, google-cloud-core, google-resumable-media,
google-cloud-storage, pyyaml, httplib2, numpy, pyarrow, grpcio, fastavro, dill,
pbr, mock, pyparsing, pydot, future, docopt, hdfs, oauth2client, avro-python3,
crcmod, grpc-google-iam-v1, google-cloud-bigtable,
proto-google-cloud-datastore-v1, monotonic, fasteners, google-apitools,
google-cloud-pubsub, google-cloud-bigquery, apache-beam, dfapp
Running setup.py develop for dfapp
Successfully installed Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1
apache-beam-2.11.0 avro-python3-1.8.2 cachetools-3.1.0 certifi-2019.3.9
chardet-3.0.4 click-7.0 crcmod-1.7 dfapp dill-0.2.9 docopt-0.6.2
fastavro-0.21.19 fasteners-0.14.1 flask-1.0.2 future-0.17.1
google-api-core-1.8.1 google-apitools-0.5.26 google-auth-1.6.3
google-cloud-bigquery-1.6.1 google-cloud-bigtable-0.31.1
google-cloud-core-0.29.1 google-cloud-pubsub-0.39.0 google-cloud-storage-1.14.0
google-resumable-media-0.3.2 googleapis-common-protos-1.5.8
grpc-google-iam-v1-0.11.4 grpcio-1.19.0 hdfs-2.2.2 httplib2-0.11.3 idna-2.8
itsdangerous-1.1.0 mock-2.0.0 monotonic-1.5 numpy-1.16.2 oauth2client-3.0.0
pbr-5.1.3 proto-google-cloud-datastore-v1-0.90.4 protobuf-3.7.0 pyarrow-0.11.1
pyasn1-0.4.5 pyasn1-modules-0.2.4 pydot-1.2.4 pyparsing-2.3.1 pytz-2018.9
pyyaml-3.13 requests-2.21.0 rsa-4.0 urllib3-1.24.1
{noformat}
I highlight here the part of interest:
{color:#ff0000}
google-cloud-bigtable 0.31.1 has requirement
google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1
which is incompatible.{color}
And pinning this quickly becomes a nightmare (looks like going in the rabbit
hole).
If i start the app, i get:
{noformat}
pkg_resources.DistributionNotFound: The 'google-cloud-core==0.28.1; extra ==
"gcp"' distribution was not found and is required by apache-beam
{noformat}
was (Author: brunoripa):
Actually i am still experiencing this issue, in a very simple package i am
creating for an article.
Basically in my {{setup.py}} i have two packages:
{quote}"flask==1.0.2",
"apache-beam[gcp]"
{quote}
but then i need to install {{google-cloud-storage}}; this causes, in a clean
virtual environment (Python3):
{noformat}
~/dev/dataflow/gcs_batch/df_server/dfapp » pip install -e .
j0k3r@Brunos-MacBook-Pro
Obtaining [file:///Users/j0k3r/dev/dataflow/gcs_batch/df_server/dfapp]
Collecting flask==1.0.2 (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl]
Collecting google-cloud-storage (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/b6/e9/06d9bb394fddbc62bb9c645f5e1c927128930a249d0c6a7491c3f31a9ff4/google_cloud_storage-1.14.0-py2.py3-none-any.whl]
Collecting apache-beam[gcp] (from dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/f7/7a/57b09ce4d6eaf7e7e192b7614a44679d6f55c11770c92f34b0ed9b59609e/apache_beam-2.11.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting itsdangerous>=0.24 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl]
Collecting Werkzeug>=0.14 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl]
Collecting click>=5.1 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl]
Collecting Jinja2>=2.10 (from flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl]
Collecting google-api-core<2.0.0dev,>=1.6.0 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/c4/71/23a234ee35117c2ed1ebd5a62ae07ef29f9f0bae9ea816b91312bad81646/google_api_core-1.8.1-py2.py3-none-any.whl]
Collecting google-cloud-core<0.30dev,>=0.29.0 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/0c/f2/3c225e7a69cb27d283b68bff867722bd066bc1858611180197f711815ea5/google_cloud_core-0.29.1-py2.py3-none-any.whl]
Collecting google-resumable-media>=0.3.1 (from
google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/e2/5d/4bc5c28c252a62efe69ed1a1561da92bd5af8eca0cdcdf8e60354fae9b29/google_resumable_media-0.3.2-py2.py3-none-any.whl]
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting httplib2<=0.11.3,>=0.8 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting protobuf<4,>=3.5.0.post1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fb/4f/b4ab63001f81ccc3ef754965070000f056b2c13fadf93ef46ac231e3086d/protobuf-3.7.0-cp37-cp37m-macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting pyarrow<0.12.0,>=0.11.1; python_version >= "3.0" or platform_system
!= "Windows" (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/50/35/25d20077820ce8bd068bc9c06570969af7d70ef32fc453b3b41fac4c45eb/pyarrow-0.11.1-cp37-cp37m-macosx_10_6_intel.whl]
Collecting grpcio<2,>=1.8 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/df/33/c0561fe7c5e235325255f46c08bd3d07f2c80824feb22d057328eff1f8b7/grpcio-1.19.0-cp37-cp37m-macosx_10_9_x86_64.whl]
Collecting fastavro<0.22,>=0.21.4 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/78/ae/2b889d19040f11b547f550e6319e6f2602a1764e19e9c84551ece163d51e/fastavro-0.21.19-cp37-cp37m-macosx_10_13_x86_64.whl]
Collecting dill<0.2.10,>=0.2.9 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
Collecting pydot<1.3,>=1.2.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting future<1.0.0,>=0.16.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting pytz>=2018.3 (from apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl]
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting oauth2client<4,>=2.0.1 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Collecting crcmod<2.0,>=1.7 (from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-cloud-bigtable==0.31.1; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/00/58/8153616835b3ff7238c657400c8fc46c44b53074b39b22260dd06345f9ed/google_cloud_bigtable-0.31.1-py2.py3-none-any.whl]
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0; extra == "gcp"
(from apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-apitools<0.5.27,>=0.5.26; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Collecting google-cloud-pubsub==0.39.0; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/fc/30/c2e6611c3ffa45816e835b016a2b40bb2bd93f05d1055f78be16a9eb2e4d/google_cloud_pubsub-0.39.0-py2.py3-none-any.whl]
Collecting google-cloud-bigquery<1.7.0,>=1.6.0; extra == "gcp" (from
apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/b7/1b/2b95f2fefddbbece38110712c225bfb5649206f4056445653bd5ca4dc86d/google_cloud_bigquery-1.6.1-py2.py3-none-any.whl]
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask==1.0.2->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/ce/c6/f000f1af136ef74e4a95e33785921c73595c5390403f102e9b231b065b7a/MarkupSafe-1.1.1-cp37-cp37m-macosx_10_6_intel.whl]
Collecting google-auth<2.0dev,>=0.4.0 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/c5/9b/ed0516cc1f7609fb0217e3057ff4f0f9f3e3ce79a369c6af4a6c5ca25664/google_auth-1.6.3-py2.py3-none-any.whl]
Collecting googleapis-common-protos!=1.5.4,<2.0dev,>=1.5.3 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Collecting requests<3.0.0dev,>=2.18.0 (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl]
Requirement already satisfied: six>=1.10.0 in
/Users/j0k3r/dev/virtualenvs/dfexample/lib/python3.7/site-packages (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
(1.12.0)
Requirement already satisfied: setuptools>=34.0.0 in
/Users/j0k3r/dev/virtualenvs/dfexample/lib/python3.7/site-packages (from
google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
(40.8.0)
Collecting numpy>=1.14 (from pyarrow<0.12.0,>=0.11.1; python_version >= "3.0"
or platform_system != "Windows"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/a6/6f/cb20ccd8f0f8581e0e090775c0e3c3e335b037818416e6fa945d924397d2/numpy-1.16.2-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
Collecting pbr>=0.11 (from
mock<3.0.0,>=1.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/09/12fe9a14237a6b7e0ba3a8d6fcf254bf4b10ec56a0185f73d651145e9222/pbr-5.1.3-py2.py3-none-any.whl]
Collecting pyparsing>=2.1.4 (from
pydot<1.3,>=1.2.0->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl]
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]->dfapp==0.1.dev0)
Collecting rsa>=3.1.4 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl]
Collecting pyasn1>=0.1.7 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl]
Collecting pyasn1-modules>=0.0.5 (from
oauth2client<4,>=2.0.1->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl]
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4 (from
google-cloud-bigtable==0.31.1; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Collecting fasteners>=0.14 (from google-apitools<0.5.27,>=0.5.26; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/3a/096c7ad18e102d4f219f5dd15951f9728ca5092a3385d2e8f79a7c1e1017/fasteners-0.14.1-py2.py3-none-any.whl]
Collecting cachetools>=2.0.0 (from
google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl]
Collecting certifi>=2017.4.17 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl]
Collecting urllib3<1.25,>=1.21.1 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl]
Collecting chardet<3.1.0,>=3.0.2 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
Collecting idna<2.9,>=2.5 (from
requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=1.6.0->google-cloud-storage->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl]
Collecting monotonic>=0.1 (from
fasteners>=0.14->google-apitools<0.5.27,>=0.5.26; extra ==
"gcp"->apache-beam[gcp]->dfapp==0.1.dev0)
Using cached
[https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl]
google-cloud-bigtable 0.31.1 has requirement
google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1
which is incompatible.
Installing collected packages: itsdangerous, Werkzeug, click, MarkupSafe,
Jinja2, flask, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, protobuf,
googleapis-common-protos, certifi, urllib3, chardet, idna, requests, pytz,
google-api-core, google-cloud-core, google-resumable-media,
google-cloud-storage, pyyaml, httplib2, numpy, pyarrow, grpcio, fastavro, dill,
pbr, mock, pyparsing, pydot, future, docopt, hdfs, oauth2client, avro-python3,
crcmod, grpc-google-iam-v1, google-cloud-bigtable,
proto-google-cloud-datastore-v1, monotonic, fasteners, google-apitools,
google-cloud-pubsub, google-cloud-bigquery, apache-beam, dfapp
Running setup.py develop for dfapp
Successfully installed Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1
apache-beam-2.11.0 avro-python3-1.8.2 cachetools-3.1.0 certifi-2019.3.9
chardet-3.0.4 click-7.0 crcmod-1.7 dfapp dill-0.2.9 docopt-0.6.2
fastavro-0.21.19 fasteners-0.14.1 flask-1.0.2 future-0.17.1
google-api-core-1.8.1 google-apitools-0.5.26 google-auth-1.6.3
google-cloud-bigquery-1.6.1 google-cloud-bigtable-0.31.1
google-cloud-core-0.29.1 google-cloud-pubsub-0.39.0 google-cloud-storage-1.14.0
google-resumable-media-0.3.2 googleapis-common-protos-1.5.8
grpc-google-iam-v1-0.11.4 grpcio-1.19.0 hdfs-2.2.2 httplib2-0.11.3 idna-2.8
itsdangerous-1.1.0 mock-2.0.0 monotonic-1.5 numpy-1.16.2 oauth2client-3.0.0
pbr-5.1.3 proto-google-cloud-datastore-v1-0.90.4 protobuf-3.7.0 pyarrow-0.11.1
pyasn1-0.4.5 pyasn1-modules-0.2.4 pydot-1.2.4 pyparsing-2.3.1 pytz-2018.9
pyyaml-3.13 requests-2.21.0 rsa-4.0 urllib3-1.24.1
{noformat}
I highlight here the part of interest:
{color:#FF0000}
google-cloud-bigtable 0.31.1 has requirement
google-cloud-core<0.29dev,>=0.28.0, but you'll have google-cloud-core 0.29.1
which is incompatible.{color}
And pinning this quickly becomes a nightmare (looks like going in the rabbit
hole).
> Consider not pinning all python dependencies, or moving them to
> requirements.txt
> --------------------------------------------------------------------------------
>
> Key: BEAM-3106
> URL: https://issues.apache.org/jira/browse/BEAM-3106
> Project: Beam
> Issue Type: Wish
> Components: build-system
> Affects Versions: 2.1.0
> Environment: python
> Reporter: Maximilian Roos
> Priority: Major
>
> Currently all python dependencies are [pinned or
> capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well
> tested dependencies, having them specified in `setup.py` forces them to an
> exact state on each install of Beam. This makes using Beam in any environment
> with other libraries nigh on impossible.
> This is particularly severe for the `gcp` dependencies, where we have
> libraries that won't work with an older version (but Beam _does_ work with an
> newer version). We have to do a bunch of gymnastics to get the correct
> versions installed because of this. Unfortunately, airflow repeats this
> practice and conflicts on a number of dependencies, adding further
> complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for
> example no libraries in numerical python do this. Here's a [discussion on
> SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
