cetingokhan opened a new issue, #58724:
URL: https://github.com/apache/airflow/issues/58724
### Apache Airflow version
3.1.3
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
I have migrated my Airflow setup from an on-premise single-node instance
(Airflow 2.x) to a Kubernetes cluster running Airflow 3.1.3 via Helm.
I am using the `PythonVirtualenvOperator` to execute tasks in an isolated
environment. Inside the callable function, I am attempting to retrieve Airflow
Variables and Connections using the new Airflow 3 Task SDK
(`airflow.sdk.Variable` and `airflow.sdk.BaseHook`).
While the DAG parses correctly, the task fails at runtime when trying to
fetch data from the Execution API / DB. The isolated virtual environment seems
unable to authenticate or reach the context required to retrieve these secrets.
This setup works if I hardcode values, but fails when using standard Airflow
retrieval methods within the virtualenv.
Errors encountered:
1. When fetching a Variable:
Error reading variable: VARIABLE_NOT_FOUND: {'message': 'Variable xxxxxxxxx
not found'} source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
2. When fetching a Connection:
Error executing query: The conn_id `xxxxxxxxx` isn't defined
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
### What you think should happen instead?
The `PythonVirtualenvOperator` should be able to access Variables and
Connections using the `airflow.sdk` methods within the virtual environment,
similar to how it works in the standard execution environment or how it worked
in previous versions (via DB access or injected context). The necessary tokens
or context for the Execution API should be propagated to the virtual
environment.
### How to reproduce
`# Prerequisites:
# 1. Create a Variable named "data_quality_containers" with a JSON value
(e.g., {"test": "value"}).
# 2. Create a Connection named "oracle_usr_lake" (or any dummy connection).
from airflow import DAG
from airflow.providers.python.operators.python import
PythonVirtualenvOperator
import pendulum
def venv_task_callable():
import json
# Airflow 3.0 Task SDK imports
from airflow.sdk import Variable, BaseHook
# 1. Read Variable
var_name = "data_quality_containers"
print(f"Reading variable: {var_name}")
try:
# Variable.get returns the value as string
var_value_str = Variable.get(var_name)
# Parse JSON
var_value = json.loads(var_value_str)
print(f"Variable value (parsed): {var_value}")
except Exception as e:
print(f"Error reading variable: {e}")
# Re-raise to show failure in logs if needed, but printing for demo
raise
# 2. Oracle Connection
conn_id = "oracle_usr_lake"
print(f"Using connection: {conn_id}")
try:
# Get the hook for the connection
hook = BaseHook.get_hook(conn_id)
# Execute query
sql = "SELECT 1 FROM DUAL"
print(f"Executing query: {sql}")
# Use get_first or similar method available in the hook
result = hook.get_first(sql)
print(f"Query result: {result}")
except Exception as e:
print(f"Error executing query: {e}")
raise
with DAG(
dag_id="venv_test_dag_repro",
start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
schedule=None,
catchup=False,
tags=["reproduction", "airflow3", "venv"],
) as dag:
task1 = PythonVirtualenvOperator(
task_id="read_var_and_query_oracle",
python_callable=venv_task_callable,
requirements=[
"apache-airflow>=3.1.3",
"apache-airflow-providers-oracle",
"dill"
],
system_site_packages=False,
inherit_env=True,
serializer="dill"
)`
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.16.0
apache-airflow-providers-apache-hdfs==4.10.3
apache-airflow-providers-celery==3.12.3
apache-airflow-providers-cncf-kubernetes==10.9.0
apache-airflow-providers-common-compat==1.8.0
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-docker==4.4.4
apache-airflow-providers-elasticsearch==6.3.4
apache-airflow-providers-fab==3.0.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.9
apache-airflow-providers-google==18.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.3
apache-airflow-providers-http==5.3.4
apache-airflow-providers-imap==3.9.2
apache-airflow-providers-microsoft-azure==12.8.0
apache-airflow-providers-microsoft-mssql==4.3.2
apache-airflow-providers-mongo==5.2.3
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.3
apache-airflow-providers-oracle==4.2.0
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-redis==4.3.2
apache-airflow-providers-samba==4.11.1
apache-airflow-providers-sendgrid==4.1.4
apache-airflow-providers-sftp==5.4.1
apache-airflow-providers-slack==9.4.0
apache-airflow-providers-smtp==2.3.1
apache-airflow-providers-snowflake==6.6.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-standard==1.9.2
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
Worker Logs;
2025-11-26T12:44:14.992634Z [info ] checking for stale bundle versions
locally [airflow.dag_processing.bundles.base] loc=base.py:222
2025-11-26T12:44:14.993053Z [info ] DAG bundles loaded: dags-folder
[airflow.dag_processing.bundles.manager.DagBundlesManager] loc=manager.py:179
2025-11-26T12:44:35.363058Z [info ] Task
execute_workload[0e973269-41a2-41ca-aded-343bfd8e00b6] received
[celery.worker.strategy] loc=strategy.py:161
2025-11-26T12:44:35.373918Z [info ]
[0e973269-41a2-41ca-aded-343bfd8e00b6] Executing workload in Celery:
token='eyJ***' ti=TaskInstance(id=UUID('019ac031-bb09-7f90-b3df-fb34895c3d92'),
dag_version_id=UUID('019abfd2-2863-7deb-abf3-5dd5a5b4353c'),
task_id='read_var_and_query_oracle', dag_id='venv_test_dag_classic',
run_id='manual__2025-11-26T12:44:26+00:00', try_number=1, map_index=-1,
pool_slots=1, queue='default', priority_weight=1, executor_config=None,
parent_context_carrier={}, context_carrier={})
dag_rel_path=PurePosixPath('venvtest_dag.py')
bundle_info=BundleInfo(name='dags-folder', version=None)
log_path='dag_id=venv_test_dag_classic/run_id=manual__2025-11-26T12:44:26+00:00/task_id=read_var_and_query_oracle/attempt=1.log'
type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
loc=celery_executor_utils.py:166
2025-11-26T12:44:35.375740Z [info ] Secrets backends loaded for worker
[supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
loc=supervisor.py:1931
INFO: 10.244.3.65:60988 - "GET
/log/dag_id%3Dvenv_test_dag_classic/run_id%3Dmanual__2025-11-26T12%3A44%3A26%2B00%3A00/task_id%3Dread_var_and_query_oracle/attempt%3D1.log
HTTP/1.1" 200 OK
INFO: 10.244.3.65:60990 - "GET
/log/dag_id%3Dvenv_test_dag_classic/run_id%3Dmanual__2025-11-26T12%3A44%3A26%2B00%3A00/task_id%3Dread_var_and_query_oracle/attempt%3D1.log
HTTP/1.1" 200 OK
2025-11-26T12:44:46.253237Z [info ] Task finished
[supervisor] duration=10.87820625997847 exit_code=0 final_state=success
loc=supervisor.py:1951 task_instance_id=019ac031-bb09-7f90-b3df-fb34895c3d92
2025-11-26T12:44:46.261329Z [info ] Task
execute_workload[0e973269-41a2-41ca-aded-343bfd8e00b6] succeeded in
10.896968158020172s: None [celery.app.trace] loc=trace.py:128
API logs;
2025-11-26T12:44:36.884002Z [info ] Task started
[airflow.api_fastapi.execution_api.routes.task_instances]
hostname=airflow-worker-1.airflow-worker.airflow.svc.cluster.local
loc=task_instances.py:199 previous_state=queued
ti_id=019ac031-bb09-7f90-b3df-fb34895c3d92
2025-11-26T12:44:36.885185Z [info ] Task instance state updated
[airflow.api_fastapi.execution_api.routes.task_instances]
loc=task_instances.py:212 rows_affected=1
ti_id=019ac031-bb09-7f90-b3df-fb34895c3d92
INFO: 10.244.4.3:46532 - "PATCH
/execution/task-instances/019ac031-bb09-7f90-b3df-fb34895c3d92/run HTTP/1.1"
200 OK
INFO: 10.244.4.3:46532 - "PUT
/execution/task-instances/019ac031-bb09-7f90-b3df-fb34895c3d92/heartbeat
HTTP/1.1" 204 No Content
2025-11-26T12:44:37.081522Z [info ] Updating RenderedTaskInstanceFields
[airflow.api_fastapi.execution_api.routes.task_instances] field_count=7
loc=task_instances.py:687 ti_id=019ac031-bb09-7f90-b3df-fb34895c3d92
INFO: 10.244.4.3:46532 - "PUT
/execution/task-instances/019ac031-bb09-7f90-b3df-fb34895c3d92/rtif HTTP/1.1"
201 Created
INFO: 10.244.3.1:41364 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO: 10.244.3.1:41362 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO: 10.244.3.69:39626 - "GET
/ui/grid/runs/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0" 200
OK
INFO: 10.244.3.69:39642 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39644 - "GET
/ui/grid/ti_summaries/venv_test_dag_classic/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39654 - "GET
/ui/grid/structure/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0"
200 OK
INFO: 10.244.3.69:39666 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/hitlDetails
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39674 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances?limit=50&offset=0&order_by=-start_date&order_by=-run_after
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39686 - "GET
/ui/grid/runs/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0" 200
OK
INFO: 10.244.3.69:39698 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39710 - "GET
/ui/grid/ti_summaries/venv_test_dag_classic/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39718 - "GET
/ui/grid/structure/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0"
200 OK
INFO: 10.244.3.69:39728 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances?limit=50&offset=0&order_by=-start_date&order_by=-run_after
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39744 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/hitlDetails
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39760 - "GET
/api/v2/dags/venv_test_dag_classic/tasks/read_var_and_query_oracle HTTP/1.0"
200 OK
INFO: 10.244.3.69:39756 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/-1
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39750 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/hitlDetails?task_id=read_var_and_query_oracle
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39766 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/logs/1?map_index=-1
HTTP/1.0" 200 OK
INFO: 10.244.4.3:33222 - "PUT
/execution/task-instances/019ac031-bb09-7f90-b3df-fb34895c3d92/heartbeat
HTTP/1.1" 204 No Content
INFO: 10.244.3.69:39778 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39794 - "GET
/ui/grid/ti_summaries/venv_test_dag_classic/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39800 - "GET
/ui/grid/runs/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0" 200
OK
INFO: 10.244.3.69:39806 - "GET
/ui/grid/structure/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0"
200 OK
INFO: 10.244.3.69:39818 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/-1
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39824 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/hitlDetails?task_id=read_var_and_query_oracle
HTTP/1.0" 200 OK
INFO: 10.244.3.69:39838 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/logs/1?map_index=-1
HTTP/1.0" 200 OK
2025-11-26T12:44:47.732893Z [info ] Task instance state updated
[airflow.api_fastapi.execution_api.routes.task_instances]
loc=task_instances.py:424 new_state=success rows_affected=1
ti_id=019ac031-bb09-7f90-b3df-fb34895c3d92
INFO: 10.244.4.3:33222 - "PATCH
/execution/task-instances/019ac031-bb09-7f90-b3df-fb34895c3d92/state HTTP/1.1"
204 No Content
INFO: 10.244.3.1:41684 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO: 10.244.3.1:41694 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO: 10.244.3.69:52852 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52860 - "GET
/ui/grid/ti_summaries/venv_test_dag_classic/manual__2025-11-26T12%3A44%3A26%2B00%3A00
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52872 - "GET
/ui/grid/runs/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0" 200
OK
INFO: 10.244.3.69:52878 - "GET
/ui/grid/structure/venv_test_dag_classic?limit=10&order_by=-run_after HTTP/1.0"
200 OK
INFO: 10.244.3.69:52894 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/-1
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52904 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/hitlDetails?task_id=read_var_and_query_oracle
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52920 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/logs/1?map_index=-1
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52922 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/logs/1?map_index=-1
HTTP/1.0" 200 OK
INFO: 10.244.3.69:52924 - "GET
/api/v2/dags/venv_test_dag_classic/dagRuns/manual__2025-11-26T12%3A44%3A26%2B00%3A00/taskInstances/read_var_and_query_oracle/logs/1?map_index=-1
HTTP/1.0" 200 OK
DAG Tasks Logs from UI:
[2025-11-26 15:44:35] INFO - DAG bundles loaded: dags-folder
source=airflow.dag_processing.bundles.manager.DagBundlesManager
loc=manager.py:179
[2025-11-26 15:44:35] INFO - Filling up the DagBag from
/opt/airflow/dags/venvtest_dag.py source=airflow.models.dagbag.DagBag
loc=dagbag.py:593
[2025-11-26 15:44:35] INFO - operators.test_operator module loaded
successfully! source=task.stdout
[2025-11-26 15:44:35] INFO - Executing cmd: uv venv --allow-existing --seed
--python python --system-site-packages /tmp/venvpz7mrfgg
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:147
[2025-11-26 15:44:35] INFO - Output:
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:157
[2025-11-26 15:44:35] INFO - Using CPython 3.12.12 interpreter at:
/usr/python/bin/python
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:35] INFO - Creating virtual environment with seed packages
at: /tmp/venvpz7mrfgg source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:35] INFO - + pip==25.3
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:35] INFO - Executing cmd: uv pip install --python
/tmp/venvpz7mrfgg/bin/python -r /tmp/venvpz7mrfgg/requirements.txt
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:147
[2025-11-26 15:44:35] INFO - Output:
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:157
[2025-11-26 15:44:35] INFO - Using Python 3.12.12 environment at:
/tmp/venvpz7mrfgg source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:36] INFO - Resolved 126 packages in 422ms
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:36] INFO - Installed 126 packages in 179ms
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:36] INFO - Executing cmd: /tmp/venvpz7mrfgg/bin/python
/tmp/venv-call06vwq95h/script.py /tmp/venv-call06vwq95h/script.in
/tmp/venv-call06vwq95h/script.out /tmp/venv-call06vwq95h/string_args.txt
/tmp/venv-call06vwq95h/termination.log
/tmp/venv-call06vwq95h/airflow_context.json
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:147
[2025-11-26 15:44:36] INFO - Output:
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:157
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.344299Z [info ] Reading
variable: data_quality_containers [airflow.task] loc=script.py:41
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.395090Z [error ] Error
reading variable: VARIABLE_NOT_FOUND: {'message': 'Variable
data_quality_containers not found'} [airflow.task] loc=script.py:51
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.395238Z [info ] Using
connection: oracle_usr_lake [airflow.task] loc=script.py:55
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.395334Z [info ]
Executing query: SELECT 1 FROM DUAL [airflow.task] loc=script.py:62
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.398810Z [error ] Error
executing query via Hook: The conn_id `oracle_usr_lake` isn't defined
[airflow.task] loc=script.py:68
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.398932Z [info ]
Attempting manual connection fallback... [airflow.task] loc=script.py:69
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.489778Z [info ]
Executing query manually: SELECT 1 FROM DUAL [airflow.task] loc=script.py:80
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - 2025-11-26T12:44:45.491045Z [info ] Query
result (manual): (1,) [airflow.task] loc=script.py:83
source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:45] INFO - operators.test_operator module loaded
successfully! source=airflow.providers.standard.utils.python_virtualenv
loc=python_virtualenv.py:161
[2025-11-26 15:44:46] WARNING - Fail to delete
/opt/airflow/tmp/venvpz7mrfgg. The directory does not exist.
source=airflow.task.operators.airflow.providers.standard.operators.python.PythonVirtualenvOperator
loc=python.py:869
[2025-11-26 15:44:46] INFO - Done. Returned value was: None
source=airflow.task.operators.airflow.providers.standard.operators.python.PythonVirtualenvOperator
loc=python.py:218
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
