potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r298472784
##########
File path: Dockerfile
##########
@@ -292,28 +320,32 @@ COPY --chown=airflow:airflow airflow/version.py
${AIRFLOW_SOURCES}/airflow/versi
COPY --chown=airflow:airflow airflow/__init__.py
${AIRFLOW_SOURCES}/airflow/__init__.py
COPY --chown=airflow:airflow airflow/bin/airflow
${AIRFLOW_SOURCES}/airflow/bin/airflow
-# Airflow Extras installed
-ARG AIRFLOW_EXTRAS="all"
-ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
-RUN echo "Installing with extras: ${AIRFLOW_EXTRAS}."
-
-# First install only dependencies but no Apache Airflow itself
-# This way regular changes in sources of Airflow will not trigger
reinstallation of all dependencies
-# And this Docker layer will be reused between builds.
-RUN pip install --no-use-pep517 -e ".[${AIRFLOW_EXTRAS}]"
+# First upgrade everything to latest version and then run pip install -e
.[extras] to downgrade the
+# dependencies which should not be the latest ones according to setup.py
+RUN pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 |
xargs pip install -U && \
Review comment:
Yeah. I will try to explain it a bit better in the comment. Basically what I
am trying to do is to upgrade to the latest version of all dependencies that
are still matching the setup.py limitations. There is no direct way to do it in
pip (`pip install --upgrade -e .[all]` does not work as expected). So what I do
instead is to upgrade all installed and outdated dependencies to latest version
and then run `pip install -e .[all]` that will downgrade those dependencies
that are not meeting setup.py limitations. This way we have at the end an
equivalent of fresh install of latest possible dependencies as if installed
today from the scratch but we build it on top of earlier requirements we
already have (thus - having very fast 'pip install step')
Example:
1) We have `moto==1.3.5` and `sshtunnel>=0.1.4,<0.2` in setup.py
2) In our virtualenv we have: `[email protected]` and [email protected]`
2) We run `pip install --upgrade moto sshtunnel` (both have newer versions
in PIP
3) Moto is installed dragging upgrades of few other packages:
```
Installing collected packages: botocore, s3transfer, boto3,
aws-sam-translator, jsonpointer, jsonpatch, cfn-lint, jsondiff, moto
Found existing installation: botocore 1.10.84
Uninstalling botocore-1.10.84:
Successfully uninstalled botocore-1.10.84
Found existing installation: s3transfer 0.1.13
Uninstalling s3transfer-0.1.13:
Successfully uninstalled s3transfer-0.1.13
Found existing installation: boto3 1.7.84
Uninstalling boto3-1.7.84:
Successfully uninstalled boto3-1.7.84
Found existing installation: jsondiff 1.1.1
Uninstalling jsondiff-1.1.1:
Successfully uninstalled jsondiff-1.1.1
Found existing installation: moto 1.3.5
Uninstalling moto-1.3.5:
Successfully uninstalled moto-1.3.5
Successfully installed aws-sam-translator-1.12.0 boto3-1.9.179
botocore-1.12.179 cfn-lint-0.22.0 jsondiff-1.1.2 jsonpatch-1.23 jsonpointer-2.0
moto-1.3.8 s3transfer-0.2.1
```
4) sshtunnel is upgraded to 0.1.5
5) We run `pip install -e .[all]
6) It downgrades moto to 1.3.5 and downgrades few packages that are not
compatible:
```
Installing collected packages: botocore, jsondiff, s3transfer, boto3, moto,
apache-airflow
Found existing installation: botocore 1.12.179
Uninstalling botocore-1.12.179:
Successfully uninstalled botocore-1.12.179
Found existing installation: jsondiff 1.1.2
Uninstalling jsondiff-1.1.2:
Successfully uninstalled jsondiff-1.1.2
Found existing installation: s3transfer 0.2.1
Uninstalling s3transfer-0.2.1:
Successfully uninstalled s3transfer-0.2.1
Found existing installation: boto3 1.9.179
Uninstalling boto3-1.9.179:
Successfully uninstalled boto3-1.9.179
Found existing installation: moto 1.3.8
Uninstalling moto-1.3.8:
Successfully uninstalled moto-1.3.8
Found existing installation: apache-airflow 2.0.0.dev0
Uninstalling apache-airflow-2.0.0.dev0:
Successfully uninstalled apache-airflow-2.0.0.dev0
Running setup.py develop for apache-airflow
Successfully installed apache-airflow boto3-1.7.84 botocore-1.10.84
jsondiff-1.1.1 moto-1.3.5 s3transfer-0.1.13
```
7) This way we updated sshtunnel to latest version compatible with setup.py
and we kept moto at 1.3.5 - it should be equivalent of running `pip install -e
.[all]` in a fresh virtualenv.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
