melugoyal opened a new issue, #48719:
URL: https://github.com/apache/airflow/issues/48719
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
when running a task with airflow 3, a 60-second sleep task failed 28 seconds
in because after a few successful task heartbeats, it got a 409 on its next
task heartbeat causing the task to fail.
this is the log from the task sdk indicating that the heartbeat was rejected
because the task was in the queued state:
```
{
detail: {
detail: {
current_state: queued
message: TI is no longer in the running state and task should
terminate
reason: not_running
}
}
event: Server indicated the task shouldn't be running anymore
level: error
logger: supervisor
status_code: 409
timestamp: 2025-04-02T22:32:29.440322Z
}
```
but all previous logs from the heartbeat endpoint indicate the task was
heartbeating with the "running" state:
```
[2025-04-02T22:32:03.360+0000] {task_instances.py:481} DEBUG - Task with
running state heartbeated
[2025-04-02T22:32:08.547+0000] {task_instances.py:481} DEBUG - Task with
running state heartbeated
[2025-04-02T22:32:13.639+0000] {task_instances.py:481} DEBUG - Task with
running state heartbeated
[2025-04-02T22:32:18.928+0000] {task_instances.py:481} DEBUG - Task with
running state heartbeated
[2025-04-02T22:32:24.173+0000] {task_instances.py:481} DEBUG - Task with
running state heartbeated
```
### What you think should happen instead?
the task should have successfully heartbeated and continued its execution to
completion.
### How to reproduce
i'd imagine this is very hard to reproduce, it only happened once out of
many many dag/task runs. not aware of any specific reason this happened at this
point.
### Operating System
linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
