Taragolis commented on issue #37136: URL: https://github.com/apache/airflow/issues/37136#issuecomment-1923567769
You should use `@task.external_python` instead, the function which you use is a private https://github.com/apache/airflow/blob/3904206b69428525db31ff7813daa0322f7b83e8/airflow/decorators/external_python.py#L43-L49 I can't reproduce it locally on Airflow 2.8 it works as expected ```python from datetime import datetime, timezone from pathlib import Path from airflow import DAG from airflow.decorators import task, external_python_task ANOTHER_VENV = Path("/Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python") assert ANOTHER_VENV.exists() assert ANOTHER_VENV.is_file() @task.external_python(python=str(ANOTHER_VENV), expect_airflow=False) def task_runner(): import sys if sys.version_info < (3, 10): from importlib_metadata import version as metadata_version else: from importlib.metadata import version as metadata_version try: ver = metadata_version("apache-airflow") except ModuleNotFoundError: print("`apache-airflow` package not found") else: print(f"found `apache-airflow` package version {ver}") return sys.executable with DAG( dag_id="issue_37136", start_date=datetime(2024, 1, 1, tzinfo=timezone.utc), schedule=None, tags=["venv", "37136"], ): task_runner() ``` ```console [2024-02-02, 10:57:31 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [queued]> [2024-02-02, 10:57:31 UTC] {taskinstance.py:1957} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [queued]> [2024-02-02, 10:57:31 UTC] {taskinstance.py:2171} INFO - Starting attempt 1 of 1 [2024-02-02, 10:57:31 UTC] {taskinstance.py:2192} INFO - Executing <Task(_PythonExternalDecoratedOperator): task_runner> on 2024-02-02 10:57:28.849895+00:00 [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:60} INFO - Started process 35155 to run task [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'issue_37136', 'task_runner', 'manual__2024-02-02T10:57:28.849895+00:00', '--job-id', '17', '--raw', '--subdir', 'DAGS_FOLDER/foo.py', '--cfg-path', '/var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/tmp8tbefl9b'] [2024-02-02, 10:57:31 UTC] {standard_task_runner.py:88} INFO - Job 17: Subtask task_runner [2024-02-02, 10:57:31 UTC] {task_command.py:423} INFO - Running <TaskInstance: issue_37136.task_runner manual__2024-02-02T10:57:28.849895+00:00 [running]> on host andreys-mbp.lan [2024-02-02, 10:57:31 UTC] {taskinstance.py:2481} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='issue_37136' AIRFLOW_CTX_TASK_ID='task_runner' AIRFLOW_CTX_EXECUTION_DATE='2024-02-02T10:57:28.849895+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-02T10:57:28.849895+00:00' [2024-02-02, 10:57:31 UTC] {process_utils.py:182} INFO - Executing cmd: /Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.py /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.in /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/script.out /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/string_args.txt /var/folders/g7/vy97v5m51_v9hdpf_qxqzqvr0000gn/T/venv-calld6yc0zua/termination.log [2024-02-02, 10:57:31 UTC] {process_utils.py:186} INFO - Output: [2024-02-02, 10:57:31 UTC] {process_utils.py:190} INFO - `apache-airflow` package not found [2024-02-02, 10:57:31 UTC] {python.py:201} INFO - Done. Returned value was: /Users/taragolis/.pyenv/versions/llama-runtime-py311/bin/python [2024-02-02, 10:57:31 UTC] {taskinstance.py:1138} INFO - Marking task as SUCCESS. dag_id=issue_37136, task_id=task_runner, execution_date=20240202T105728, start_date=20240202T105731, end_date=20240202T105731 [2024-02-02, 10:57:31 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 0 [2024-02-02, 10:57:31 UTC] {taskinstance.py:3281} INFO - 0 downstream tasks scheduled from follow-on schedule check ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
