lihan commented on issue #36920:
URL: https://github.com/apache/airflow/issues/36920#issuecomment-1979547584
Hi, we are heavily affected by this. We are on 2.7.2.
Some observations:
* The dag lock up happens to a lot dags, but probably half of them never got
affected. The other half has the problems.
* When the affected task gets scheduled, I can see
```
{scheduler_job_runner.py:686} INFO - Received executor event with state
failed for task instance TaskInstanceKey(dag_id='dag_id', task_id='task_id',
run_id='scheduled__2024-02-02T00:00:00+00:00', try_number=5, map_index=-1)
{scheduler_job_runner.py:723} INFO - TaskInstance Finished: dag_id=dag_id,
task_id=task_id, run_id=scheduled__2024-02-02T00:00:00+00:00, map_index=-1,
run_start_date=None, run_end_date=2024-03-04 11:58:07.112417+00:00,
run_duration=None, state=queued, executor_state=failed, try_number=5,
max_tries=6, job_id=None, pool=bigquery_pool, queue=default,
priority_weight=10, operator=CustomBigQueryOperator, queued_dttm=2024-03-05
20:00:33.986480+00:00, queued_by_job_id=22144438, pid=None
{scheduler_job_runner.py:771} ERROR - Executor reports task instance
<TaskInstance: dag_id.task_id scheduled__2024-02-02T00:00:00+00:00 [queued]>
finished (failed) although the task says it's queued. (Info: None) Was the task
killed externally?
```
The key part is `Was the task killed externally?`, this happens a lot of
times. The task was never ran by the `KubernetesExecutor`.
And in the Airflow task logging view, we see
```airflow-worker-7b84c4799d-45r9t
*** No logs found in GCS; ti=%s <TaskInstance: dag_id.task_id
scheduled__2024-02-02T00:00:00+00:00 [up_for_retry]>
```
This issue was firstly reported on 2nd February 2024, there was no Airflow
upgrade associated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
