ashb commented on code in PR #22976:
URL: https://github.com/apache/airflow/pull/22976#discussion_r892422290
##########
airflow/executors/kubernetes_executor.py:
##########
@@ -689,9 +689,14 @@ def _change_state(self, key: TaskInstanceKey, state:
Optional[str], pod_id: str,
self.event_buffer[key] = state, None
def try_adopt_task_instances(self, tis: List[TaskInstance]) ->
List[TaskInstance]:
- tis_to_flush = [ti for ti in tis if not ti.queued_by_job_id]
- scheduler_job_ids = {ti.queued_by_job_id for ti in tis}
- pod_ids = {ti.key: ti for ti in tis if ti.queued_by_job_id}
+ scheduler_job_ids = {ti.queued_by_job_id for ti in tis if
ti.queued_by_job_id}
+
+ # Tasks triggered through API will have no ti.queued_by_job_id
+ # and their pod will have label 'airflow-worker=manual'
+ if any(ti for ti in tis if not ti.queued_by_job_id):
+ scheduler_job_ids.add('manual')
Review Comment:
This introduces a new race condition with multiple schedulers:
If there is a manually launched task, and two schedulers both try to adopt
this task, one of them will win (as in the label will be changed) but both
schedulers will end up adding the pod to their `running` list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
