benbole opened a new issue, #34688:
URL: https://github.com/apache/airflow/issues/34688
### Apache Airflow version
2.7.1
### What happened
Some modules I have written have dependency conflicts with Airflow's
required Python environment, so I decided to use `ExternalPythonOperator`s and
venvs as prescribed in the Airflow documentation. I am able to build the venv,
build the Airflow Docker containers, and import all DAGs without issue, but
when the Airflow executes the `ExternalPythonOperator` which attempts to import
an Airflow library (such as `PostgresHook`) it results in error.
Some additional information that may be helpful:
- I am able to successfully import Airflow libraries within the venv's
Python Interpreter when executing it via command line
- The DAG is able to successfully run when replacing the
`ExternalPythonOperator` with a typical `@task` `PythonOperator`
### What you think should happen instead
_No response_
### How to reproduce
Below is the simplest example DAG/ExternalPythonOperator I was able to
produce which still results in error. It is scrapped together based on a
simplified version of my DAG, so there may be some unnecessary imports.
The DAG:
```
from __future__ import annotations
import os
import yaml
from datetime import datetime
from airflow.models import DAG
from airflow.decorators import dag, task
DAG_ID = "example_break"
@task.external_python(python='/opt/airflow/python_venvs/operators_venv/bin/python',
expect_airflow=True)
def this_should_break():
from operators.break_example import example_break
return example_break()
@dag(dag_id=DAG_ID,
schedule=None,
start_date=datetime(2021, 1, 1),
catchup=False,
tags=["example"],
)
def ExampleBreak():
this_should_break()
ExampleBreak()
```
The task:
```
from airflow.providers.postgres.hooks.postgres import PostgresHook #You can
substitute any Airflow library here
def example_break():
print('This should fail')
```
### Operating System
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux"
VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian
HOME_URL="https://www.debian.org/"; SUPPORT_URL="https://www.debian.org/support";
BUG_REPORT_URL="https://bugs.debian.org/";
### Versions of Apache Airflow Providers
Attached are the pip list for both my Airflow environment and my venv.
[venv_pip_list.txt](https://github.com/apache/airflow/files/12766222/venv_pip_list.txt)
[airflow_docker_pip_list.txt](https://github.com/apache/airflow/files/12766223/airflow_docker_pip_list.txt)
### Deployment
Docker-Compose
### Deployment details
I have customized the default docker-compose in order to use a custom
Dockerfile for importing needed libraries. You can find both the docker-compose
and the Dockerfile attached here.
[Dockerfile.txt](https://github.com/apache/airflow/files/12766240/Dockerfile.txt)
[docker-compose.txt](https://github.com/apache/airflow/files/12766248/docker-compose.txt)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
