vlieven opened a new issue, #36809:
URL: https://github.com/apache/airflow/issues/36809
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.7.3
### What happened?
If the `python_callable` used by a ShortCircuitOperator does not properly
return a boolean, but instead crashes, the retry and downstream behavior is not
deterministic. Either:
- the task properly retries, succeeds and downstream tasks are triggered
- the task does not properly retry, downstream tasks are marked as
upstream_failed
- the task does retry, is not marked as success, but downstream tasks are
triggered
### What you think should happen instead?
In case the callable crashes, it should be treated as a failure, and retry
logic should be properly executed until the task completes successfully.
### How to reproduce
As the behavior is not deterministic, I've created 10 parallel flows to
demonstrate the issue. A crash is simulated by `exit(-1)`.
This DAG file should demonstrate the broken behavior:
```python
from airflow import DAG
from airflow.models.taskinstance import TaskInstance
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import ShortCircuitOperator
from datetime import datetime, timedelta
default_args = {
"owner": "Test",
"depends_on_past": False,
"start_date": datetime.now() - timedelta(days=2),
"retries": 1,
"retry_delay": timedelta(seconds=5),
}
def python_callable(ti: TaskInstance):
if ti.try_number % 2 == 1:
exit(-1)
else:
return True
with DAG(
"test_shortcircuit",
default_args=default_args,
schedule_interval="@daily",
) as test_dag:
for i in range(10):
ShortCircuitOperator(
task_id=f"short-{i}", python_callable=python_callable,
) >> EmptyOperator(
task_id=f"empty-{i}"
)
```
The resulting grid view is the following:
<img width="514" alt="Screenshot 2024-01-16 at 09 14 08"
src="https://github.com/apache/airflow/assets/16362535/69fa9e39-ec0f-4daf-a7e7-163b670fc65c">
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
This is using only the built-in operators
### Deployment
Other Docker-based deployment
### Deployment details
Running on a Kubernetes cluster
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]