GitHub user Neon-Face created a discussion: [Bug] `standard.operators.python`
fails the task randomly
### Apache Airflow version
3.0.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Since I want to pass Airflow Variables from a regular `@task` to a
`@task.virtualenv`, I first created a normal task to fetch the variables and
then passed the resulting dictionary to the virtualenv task.
Here's the DAG code:
**Path:** `dags/test/dag_test.py`
(*Please note the file paths — I'm not entirely sure if separating DAG and task
definitions like this is supported.*)
``` python
from airflow.sdk import dag, Param
@dag(
dag_id="test",
params={
"P": Param("HI", type="string")
}
)
def test():
from utils.common_tools import clear_dag_xcom, load_vars_for_venv
from utils.test import virtual_task
variables = load_vars_for_venv(var_list=["DATASETTE_DB_SECRET"])
t = virtual_task(variables=variables)
c = clear_dag_xcom()
t >> c
test()
```
here is my task code:
**Path**: `utils/test.py`
```python
from airflow.sdk import task
@task.virtualenv(
python_version = "3.9",
requirements="requirements_test.txt",
system_site_packages = True,
venv_cache_path="venvs/test",
inherit_env = True
)
def virtual_task(variables:dict, logical_date = None, params = None):
print(variables)
print(f"This is param: {params} !!!!!")
print(f"This is logical_date: {logical_date} !!!!!")
return
```
and here is my common tasks (to improve reusability)
**Path**: `utils/common_tools.py`
```python
from airflow.sdk import task
@task(task_id="load_vars_for_venv")
def load_vars_for_venv(var_list:list):
from airflow.sdk import Variable
variables = {}
for variable in var_list:
variables[variable] = Variable.get(variable)
return variables
@task(task_id = "clear_dag_xcom")
def clear_dag_xcom(run_id = None): # I also created this to clean XCOM data if
the dag succeeded
import os
from airflow.sdk import Variable
airflow_db_path = Variable.get("AIRFLOW_DB_SECRET")
sql = f"DELETE FROM xcom WHERE run_id='{run_id}';"
print(sql)
os.system(f'sqlite3 {airflow_db_path} "{sql}"')
return
```
However, the load_vars_for_venv task is quite unstable — it works
intermittently. When it fails, there are no logs shown, making it difficult to
debug.
<img width="1920" height="998" alt="Image"
src="https://github.com/user-attachments/assets/5095c637-71fa-4a39-b398-4150a1164c7a";
/>
and the scheduler shows:
``` scheduler
2025-08-07 12:02:42 [info] Secrets backends loaded for worker [supervisor]
backend_classes=['EnvironmentVariablesBackend'] count=1
[2025-08-07T12:02:42.594+0000] {_client.py:1026} INFO - HTTP Request: PATCH
http://localhost:8080/execution/task-instances/01988469-7ada-7fee-b
011-bba401a05876/run "HTTP/1.1 409 Conflict"
2025-08-07 12:02:42 [warning] Server error [airflow.sdk.api.client]
detail={'detail': {'reason': 'invalid_state', 'message'
: 'TI was not in a state where it could be marked as running',
'previous_state': 'scheduled'}}
2025-08-07 12:02:42 [info] Process exited [supervisor]
exit_code=<Negsignal.SIGKILL: -9> pid=516354 signal_sent=SIGKILL
```
and
``` scheduler
[2025-08-07T12:02:42.611+0000] {local_executor.py:96} ERROR - uhoh
.......
File
"/home/....../Airflow/.venv/lib64/python3.9/site-packages/airflow/sdk/api/client.py",
line 110, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Server returned error
```
However, when I run the DAG using `airflow dags test test`, it consistently
succeeds and produces the expected logs.
```
[2025-08-07T12:25:13.682+0000] {process_utils.py:196} INFO -
{'DATASETTE_DB_SECRET': '......./datasette/insights.db'}
[2025-08-07T12:25:13.683+0000] {process_utils.py:196} INFO - This is param:
{'P': 'HI'} !!!!!
[2025-08-07T12:25:13.683+0000] {process_utils.py:196} INFO - This is
logical_date: 2025-08-07 12:25:10.918707+00:00 !!!!!
```
And you can see it's successful in airflow UI as well, but when I try to read
the log in airflow UI, it shows "Could not read served logs: 404 Client Error:
NOT FOUND for url......"
<img width="1920" height="999" alt="Image"
src="https://github.com/user-attachments/assets/c2ddba76-3d3b-4441-abd9-8f064e936189";
/>
**So my questions are:**
1. Why does the `load_vars_for_venv` task behave inconsistently? Sometimes it
works, other times it silently fails without any logs.
2. What are the exact differences between triggering a DAG via the **Airflow
UI** and using the CLI command `airflow dags test`? Why does the CLI test
consistently succeed, while the UI-triggered runs fail or get stuck?
Thanks for helping!
### What you think should happen instead?
variables(dict) should be passed to `@task.virtualenv` and variables can be
used in the virtualenv.
### How to reproduce
Copy the code above and use the same file structure should be able to reproduce
the problem.
### Operating System
Oracle Linux Server
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.12.0
apache-airflow-providers-common-compat==1.7.1
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.2
apache-airflow-providers-fab==2.2.1
apache-airflow-providers-http==5.3.1
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-standard==1.3.0
```
### Deployment
Other
### Deployment details
I installed locally using `uv`.
```
uv add "apache-airflow[celery,pandas,fab]==3.0.3" --constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-3.0.3/constraints-3.9.txt";
uv add ruff
uv add apache-airflow-providers-http
```
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/54534
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
