Enucatl opened a new issue, #54645: URL: https://github.com/apache/airflow/issues/54645
### Apache Airflow Provider(s) hashicorp ### Versions of Apache Airflow Providers `FROM docker.io/apache/airflow:3.0.4@sha256:6c07868196d1880d956f4b5ddf50244bc281faab39704a8940b1c1f3159a145e` ### Apache Airflow version 3.0.4 ### Operating System Ubuntu 22.04.5 LTS ### Deployment Docker-Compose ### Deployment details Full docker-compose if needed: https://github.com/Enucatl/docker-airflow/blob/main/docker-compose.yml ### What happened Connections defined in Hashicorp Vault are not available within `PythonVirtualenvOperators` ### What you think should happen instead test_connection log (no virtualenv = successful) ```log Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"] ::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_connection/attempt=1.log"] [2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager" [2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag" [2025-08-19, 08:42:12] INFO - Connection Retrieved 'stva': source="airflow.hooks.base" [2025-08-19, 08:42:12] INFO - Done. Returned value was: None: source="airflow.task.operators.airflow.providers.standard.decorators.python._PythonDecoratedOperator" [2025-08-19, 08:42:12] INFO - https://portal.stva.zh.ch/ecari-dispoweb/ui/app/init/#/conduite/prive/login: chan="stdout": source="task" ``` test_virtualenv log (fails) ```log ::group::Log message source details: sources=["/opt/airflow/logs/dag_id=test_vault_virtualenv/run_id=scheduled__2025-08-17T00:00:00+00:00/task_id=test_virtualenv/attempt=1.log"] [2025-08-19, 08:42:12] INFO - DAG bundles loaded: dags-folder: source="airflow.dag_processing.bundles.manager.DagBundlesManager" [2025-08-19, 08:42:12] INFO - Filling up the DagBag from /opt/airflow/dags/test_vault_virtualenv.py: source="airflow.models.dagbag.DagBag" [2025-08-19, 08:42:12] WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/models/variable.py:147: DeprecationWarning: Using Variable.get from `airflow.models` is deprecated.Please use `from airflow.sdk import Variable` instead warnings.warn( : source="py.warnings" [2025-08-19, 08:42:12] INFO - Python virtual environment will be cached in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Executing cmd: uv venv --allow-existing --seed --python python --system-site-packages /opt/airflow/venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Using CPython 3.12.11 interpreter at: /usr/local/bin/python: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Creating virtual environment with seed packages at: venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - If the cache and target directories are on different filesystems, hardlinking may not be supported.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - + pip==25.2: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Executing cmd: uv pip install --python /opt/airflow/venv/venv-aacc1679/bin/python -r /opt/airflow/venv/venv-aacc1679/requirements.txt: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - warning: Requirements file `venv/venv-aacc1679/requirements.txt` does not contain any dependencies: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Using Python 3.12.11 environment at: venv/venv-aacc1679: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Audited in 62ms: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - New Python virtual environment created in /opt/airflow/venv/venv-aacc1679: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Use 'pickle' as serializer.: source="airflow.task.operators.airflow.providers.standard.decorators.python_virtualenv._PythonVirtualenvDecoratedOperator" [2025-08-19, 08:42:12] INFO - Executing cmd: /opt/airflow/venv/venv-aacc1679/bin/python /tmp/venv-callpa921oow/script.py /tmp/venv-callpa921oow/script.in /tmp/venv-callpa921oow/script.out /tmp/venv-callpa921oow/string_args.txt /tmp/venv-callpa921oow/termination.log /tmp/venv-callpa921oow/airflow_context.json: source="airflow.utils.process_utils" [2025-08-19, 08:42:12] INFO - Output:: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - Traceback (most recent call last):: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/tmp/venv-callpa921oow/script.py", line 54, in <module>: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - res = test_virtualenv(*arg_dict["args"], **arg_dict["kwargs"]): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/tmp/venv-callpa921oow/script.py", line 26, in test_virtualenv: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - connection = BaseHook.get_connection("stva"): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/home/airflow/.local/lib/python3.12/site-packages/airflow/hooks/base.py", line 64, in get_connection: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - conn = Connection.get_connection_from_secrets(conn_id): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 507, in get_connection_from_secrets: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined"): source="airflow.utils.process_utils" [2025-08-19, 08:42:15] INFO - airflow.exceptions.AirflowNotFoundException: The conn_id `stva` isn't defined: source="airflow.utils.process_utils" [2025-08-19, 08:42:15] ERROR - Task failed with exception: source="task" AirflowException: Process returned non-zero exit status 1. The conn_id `stva` isn't defined File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 867 in run File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1159 in _execute_task File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 251 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 490 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 397 in wrapper File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 216 in execute File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 844 in execute_callable File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 576 in _execute_python_callable_in_subprocess ``` ### How to reproduce Minimum reproducible DAG: ```python from datetime import datetime, timedelta from airflow import DAG from airflow.decorators import task from airflow.hooks.base import BaseHook default_args = { "depends_on_past": False, "retries": 0, "retry_delay": timedelta(minutes=30), "venv_cache_path": "/opt/airflow/venv", } with DAG( "test_vault_virtualenv", default_args=default_args, description="Test vault access for connections", schedule="@once", start_date=datetime(2025, 8, 17), ) as dag: @task def test_connection(): connection = BaseHook.get_connection("stva") print(connection.host) @task.virtualenv( task_id="test_virtualenv", requirements=[], ) def test_virtualenv(logical_date): from airflow.hooks.base import BaseHook connection = BaseHook.get_connection("stva") print(connection.host) test_connection() test_virtualenv() ``` ### Anything else _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
