Raul824 opened a new issue, #36039:
URL: https://github.com/apache/airflow/issues/36039
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
We have setup retry in case of failures and disconnects on individual task.
Some tasks are being reran post completion.
If a task is success it should not attempt to rerun it again.
At 8:05 the task was marked success.
[2023-12-04, 08:05:23 AEDT] {taskinstance.py:1350} INFO - Marking task as
SUCCESS. dag_id=UDPPRDAU_ODS_KEY_SCD_SCF_06,
task_id=SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL,
execution_date=20231202T125900, start_date=20231203T210522,
end_date=20231203T210523
But At 8:42 it again reran the same task and there are no clear events on
this task in audit log.
*** Found local files:
*** *
/opt/airflow/logs/dag_id=UDPPRDAU_ODS_KEY_SCD_SCF_06/run_id=manual__2023-12-02T12:59:00+00:00/task_id=SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL/attempt=3.log
[2023-12-04, 08:42:20 AEDT] {taskinstance.py:1097} INFO - Dependencies not
met for <TaskInstance:
UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL
manual__2023-12-02T12:59:00+00:00 [success]>, dependency 'Task Instance State'
FAILED: Task is in the 'success' state.
[2023-12-04, 08:42:20 AEDT] {local_task_job_runner.py:154} INFO - Task is
not able to be run
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1103} INFO - Dependencies all
met for dep_context=non-requeueable deps ti=<TaskInstance:
UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL
manual__2023-12-02T12:59:00+00:00 [queued]>
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1103} INFO - Dependencies all
met for dep_context=requeueable deps ti=<TaskInstance:
UDPPRDAU_ODS_KEY_SCD_SCF_06.SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL
manual__2023-12-02T12:59:00+00:00 [queued]>
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1308} INFO - Starting attempt 3
of 501
[2023-12-04, 09:45:29 AEDT] {taskinstance.py:1327} INFO - Executing
<Task(PythonOperator):
SSOT_ODS_STG_GL_DTL_DATA_PAY_DTL.ODS_GL_DTL_DATA_PAY_DTL> on 2023-12-02
12:59:00+00:00
[2023-12-04, 09:45:29 AEDT] {standard_task_runner.py:57} INFO - Started
process 1128 to run task
My understanding was that retries comes into picture only in case of
failures, which working fine in most of the tasks but it randomly reruns the
tasks even after success.
### What you think should happen instead
Once the task is success, airflow should not retry the task.
### How to reproduce
Set up_for_retry for all tasks and run 200 tasks at the same time.
### Operating System
Azure Kubernetes Services
### Versions of Apache Airflow Providers
Airflow 2.6.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
