eejbyfeldt opened a new issue #13808:
URL: https://github.com/apache/airflow/issues/13808
**Apache Airflow version**:
2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
1.15.7
**Environment**:
- **Cloud provider or hardware configuration**: AWS
**What happened**:
We run two schedulers that were restarted simultaneously.
The first scheduler correctly adopts these orphaned tasks:
```
[2021-01-20 16:28:06,744] {kubernetes_executor.py:626} INFO - attempting to
adopt pod dag1sometask-bcf1adde250
[2021-01-20 16:28:06,762] {kubernetes_executor.py:147} INFO - Event:
dag1sometask-bcf1adde250 had an event of type ADDED
[2021-01-20 16:28:06,762] {kubernetes_executor.py:210} INFO - Event:
dag1sometask-bcf1adde250 is Running
```
Then 15 minutes later the second scheduler goes and incorrectly reset the
state for this TaskInstance:
```
[2021-01-20 16:43:10,383] {scheduler_job.py:1764} INFO - Marked 1
SchedulerJob instances as failed
[2021-01-20 16:43:10,591] {scheduler_job.py:1805} INFO - Reset the following
1 orphaned TaskInstances:
<TaskInstance: dag1.sometask 2021-01-19 01:00:00+00:00 [running]>
```
**What you expected to happen**:
One of the schedulers adopts the orphaned task and the other does nothing
with this task.
**How to reproduce it**:
Use 2 schedulers and have tasks running using the KubernetesExecutor.
Restart both schedulers simultaniously.
Wait for some task to fail due to external state change of task instance.
**Anything else we need to know**:
The following configurations was used for the scheduler:
```
scheduler_heartbeat_sec = 15
scheduler_health_check_threshold = 30
orphaned_tasks_check_interval = 300.0
```
We have not seen the issue every time we restart the scheulers, but it has
been happened more than once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
