Manuel Rodríguez Guimeráns created AIRFLOW-4098:
---------------------------------------------------
Summary: gcp_api dependency conflicts
Key: AIRFLOW-4098
URL: https://issues.apache.org/jira/browse/AIRFLOW-4098
Project: Apache Airflow
Issue Type: Bug
Components: gcp
Affects Versions: 1.10.2
Environment: Ubuntu 16.04.6 LTS
Python 2.7 / Python 3.5 / Python 3.6
Reporter: Manuel Rodríguez Guimeráns
The {{gcp_api}} extra contains inconsistencies in its subdependencies. When
installed with {{pip}} it completes with a warning, but {{pipenv}} and
{{poetry}} are not so forgiving and it cannot be installed at all.
Note that on the master branch as of 419e30 (where {{gcp_api}} is now called
just {{gcp}}) there are still dependency errors as well.
h3. pip with python 2.7
{code:java}
$ mkvirtualenv -p /usr/bin/python2.7 airflow_pip
...
$ pip --version
pip 19.0.3 from
/home/manu/.local/share/virtualenvs/airflow_pip/local/lib/python2.7/site-packages/pip
(python 2.7)
$ SLUGIFY_USES_TEXT_UNIDECODE=yes pip install 'apache-airflow[gcp_api]==1.10.2'
...
google-cloud-bigquery 1.10.0 has requirement
google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 0.28.1
which is incompatible.
google-cloud-spanner 1.8.0 has requirement google-cloud-core<0.30dev,>=0.29.0,
but you'll have google-cloud-core 0.28.1 which is incompatible.
...
{code}
h3. pipenv with python 3.6
{code:java}
$ mkdir airflow_pipenv
$ cd airflow_pipenv
$ pipenv --version
pipenv, version 2018.11.26
$ pipenv --python 3.6
Creating a virtualenv for this project...
Pipfile: /tmp/airflow_pipenv/Pipfile
Using /home/manu/.pyenv/versions/3.6.6/bin/python3.6 (3.6.6) to create
virtualenv...
⠋ Creating virtual environment...Already using interpreter
/home/manu/.pyenv/versions/3.6.6/bin/python3.6
Using base prefix '/home/manu/.pyenv/versions/3.6.6'
New python executable in
/home/manu/.local/share/virtualenvs/airflow_pipenv-ePWbOumO/bin/python3.6
Also creating executable in
/home/manu/.local/share/virtualenvs/airflow_pipenv-ePWbOumO/bin/python
Installing setuptools, pip, wheel...done.
✔ Successfully created virtual environment!
Virtualenv location: /home/manu/.local/share/virtualenvs/airflow_pipenv-ePWbOumO
$ airflow_pipenv pipenv install 'apache-airflow[gcp_api]==1.10.2'
Installing apache-airflow[gcp_api]==1.10.2...
Adding apache-airflow to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (e3de4b) out of date, updating to (ca72e7)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/resolver.py",
line 69, in resolve
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise
ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version
that matches google-cloud-core<0.29dev,<0.30dev,>=0.28.0,>=0.29.0
[pipenv.exceptions.ResolutionFailure]: Tried: 0.20.0, 0.20.0, 0.21.0,
0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.24.0,
0.24.0, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.27.1,
0.27.1, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.29.0, 0.29.0, 0.29.1, 0.29.1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be
resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with $ pipenv lock --clear, then try
the original command again.
Alternatively, you can use $ pipenv install --skip-lock to bypass this
mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches
google-cloud-core<0.29dev,<0.30dev,>=0.28.0,>=0.29.0
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0,
0.23.0, 0.23.1, 0.23.1, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.26.0,
0.26.0, 0.27.0, 0.27.0, 0.27.1, 0.27.1, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.29.0,
0.29.0, 0.29.1, 0.29.1
There are incompatible versions in the resolved dependencies.
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File
"/home/manu/.pyenv/versions/3.6.6/lib/python3.6/site-packages/pipenv/utils.py",
line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise
ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version
that matches google-cloud-core<0.29dev,<0.30dev,>=0.28.0,>=0.29.0
[pipenv.exceptions.ResolutionFailure]: Tried: 0.20.0, 0.20.0, 0.21.0,
0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0, 0.23.0, 0.23.1, 0.23.1, 0.24.0,
0.24.0, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.26.0, 0.26.0, 0.27.0, 0.27.0, 0.27.1,
0.27.1, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.29.0, 0.29.0, 0.29.1, 0.29.1
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be
resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with $ pipenv lock --clear, then try
the original command again.
Alternatively, you can use $ pipenv install --skip-lock to bypass this
mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches
google-cloud-core<0.29dev,<0.30dev,>=0.28.0,>=0.29.0
Tried: 0.20.0, 0.20.0, 0.21.0, 0.21.0, 0.22.0, 0.22.0, 0.22.1, 0.22.1, 0.23.0,
0.23.0, 0.23.1, 0.23.1, 0.24.0, 0.24.0, 0.24.1, 0.24.1, 0.25.0, 0.25.0, 0.26.0,
0.26.0, 0.27.0, 0.27.0, 0.27.1, 0.27.1, 0.28.0, 0.28.0, 0.28.1, 0.28.1, 0.29.0,
0.29.0, 0.29.1, 0.29.1
There are incompatible versions in the resolved dependencies.
{code}
h3. poetry with python 3.5
{code:java}
$ pyenv shell 3.5.5
$ poetry --version
Poetry 0.12.11
$ poetry new airflow_poetry
Created package airflow_poetry in airflow_poetry
$ cd airflow_poetry
$ poetry add apache-airflow==1.10.2 -E gcp_api
Creating virtualenv airflow-poetry-py3.5 in
/home/manu/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (3.0s)
[SolverProblemError]
Because no versions of google-cloud-spanner match >1.7.1,<1.8.0 || >1.8.0
and google-cloud-spanner (1.7.1) depends on google-cloud-core
(>=0.29.0,<0.30dev), google-cloud-spanner (>=1.7.1,<1.8.0 || >1.8.0) requires
google-cloud-core (>=0.29.0,<0.30dev).
And because google-cloud-spanner (1.8.0) depends on google-cloud-core
(>=0.29.0,<0.30dev), google-cloud-spanner (>=1.7.1) requires google-cloud-core
(>=0.29.0,<0.30dev).
And because apache-airflow (1.10.2) depends on google-cloud-bigtable (0.31.0)
which depends on google-cloud-core (>=0.28.0,<0.29dev), google-cloud-spanner
(>=1.7.1) is incompatible with apache-airflow (1.10.2
).
So, because airflow-poetry depends on apache-airflow (=1.10.2) which depends on
google-cloud-spanner (>=1.7.1), version solving failed.
add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional]
[--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--]
<name> (<name>)...
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
