EricBoix opened a new issue #14189:
URL: https://github.com/apache/airflow/issues/14189
**Apache Airflow version**: 2.0.1
**Kubernetes version** : not used
**Environment**: docker engine 20.10.0 (provided by docker-desktop 3.0.1,
installed with `brew cask install docker` and then accepting the docker-desktop
GUI proposal to upgrade to 3.0.1).
- **Cloud provider or hardware configuration**: MacBook Pro
- **OS** : OSX 10.14.6 (Mojave)
- **Kernel**: ("airflow info" reports) system='Darwin', release='18.7.0'
- **Install tools**: Pypi
- **Others**:
**What happened**:
After installing airflow 2.0.1 with Pypi, I tried to run the docker examples
(as provided by the Airflow github repository) from the CLI. This failed (for
me).
**What you expected to happen**:
Apparently the docker example dags seem to be failing because of an API
mismatch between the version used in Airflow's docker-examples and the
"contemporary" version of Docker Python wrappings.
In turn, could this be due (?) to:
- the evolution of the
[airflow.operators.docker_operator](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/docker_operator/index.html?highlight=docker%20operator#module-airflow.operators.docker_operator
) to `airflow.providers.docker.operators.docker` ?
- the apparent disparition of the docker executor backend [from the list
of
executors](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#supported-backends)
? (Could it be that the Kubernetes executor is the new default desktop
executor?)
**How to reproduce it**:
Follow Airflow [quick Pypi
installation](https://airflow.apache.org/docs/apache-airflow/stable/start.html)
that boils down to:
```
$ virtualenv -p python3.8 venv # 3.9 not supported yet
$ source venv/bin/activate
(venv) pip --version # Yields 20.2.4 as required
(venv) export AIRFLOW_VERSION=2.0.1
(venv) export PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d
"." -f 1-2)"
(venv) export
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt";
(venv) pip install
"apache-airflow[docker,cncf.kubernetes]==${AIRFLOW_VERSION}" --constraint
"${CONSTRAINT_URL}"
(venv) airflow version ### Just to make sure
```
To run the docker-examples, the airflow sources are required
```bash
$ git clone -b 2.0.1 https://github.com/apache/airflow.git
$ mv airflow airflow.git
$ export AIRFLOW_REPO=`pwd`/airflow.git
$ mkdir $AIRFLOW_HOME/dags
$ cd $AIRFLOW_HOME/dags
$ ln -s $AIRFLOW_REPO/airflow/providers/docker/example_dags
./docker-example_dags
```
Proceed with running de `docker_sample`
```bash
(venv) pip install docker # Dependency should
already be satisfied
(venv) airflow db init # If not already
done
(venv) airflow webserver -D --port 8080 # Shouldn't hurt although
propably not required here (?)
(venv) airflow scheduler -D
(venv) airflow dags list | grep -i docker # Assert the docker examples
are visible
(venv) airflow dags test docker_sample now
```
where this last command will issue an error of the form
```
{taskinstance.py:1396} ERROR - API versions below 1.21 are no longer
supported by this library.
```
and further suggesting that
`airflow/venv/lib/python3.8/site-packages/airflow/providers/docker/operators/docker.py`
(line 314, in `_get_cli`) is using a deprecated API (with docker version
`20.10.0` on OSX).
If we now manually downgrade the version of the docker python wrapper
package starting from 3.7.3 (which is the version pinned by the
[CONSTRAINT_URL](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.8.txt)
that thus was installed by `pip`, refer above) down to 3.0.0 with e.g.
```
pip install docker==3.0.0
```
then one consistently gets the same API version error message.
But starting with version 2.7.0 of the docker python wrappers then the
connection to the docker daemon seems no longer possible (and should fail with
message of the form `requests.exceptions.ConnectionError: HTTPConnectionPool
[...]`).
## Install minikube/kind
The goal was/is to use docker only. This is because, on a desktop used for
debugging my
dags, Kubernetes is heavier than docker (my mileage is low) to deploy and to
use.
**Anything else we need to know**:
Nope. Seems about it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
