KarthikeyanDevendhiran commented on issue #51301:
URL: https://github.com/apache/airflow/issues/51301#issuecomment-2959899655
Hi @karenbraganz , Apologies for the delayed response. I’ve shared the
scheduler logs and Airflow metadata logs for the two tasks that were marked as
failed but did not trigger any Slack alerts. Please find the requested logs
below for your review.
**Scheduler logs**
```
`kubectl logs airflow-scheduler-65cfcf89c9-w8xjc -n airflow | grep
"TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00'"
Defaulted container "scheduler" out of: scheduler, git-sync,
scheduler-log-groomer, wait-for-airflow-migrations (init), git-sync-init (init)
[2025-06-09T10:00:00.476+0000] {scheduler_job_runner.py:692} INFO - Sending
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) to
KubernetesExecutor with priority 1 and queue default
[2025-06-09T10:00:00.485+0000] {kubernetes_executor.py:352} INFO - Add task
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) with
command ['airflow', 'tasks', 'run', 'dag_name', 'task_name',
'scheduled__2025-06-09T08:00:00+00:00', '--local', '--subdir',
'DAGS_FOLDER/folder_name.py']
[2025-06-09T10:00:05.201+0000] {scheduler_job_runner.py:776} INFO - Received
executor event with state queued for task instance
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1)
[2025-06-09T10:00:12.165+0000] {kubernetes_executor_utils.py:425} INFO -
Creating kubernetes pod for job is TaskInstanceKey(dag_id='dag_name',
task_id='task_name', run_id='scheduled__2025-06-09T08:00:00+00:00',
try_number=1, map_index=-1), with pod name dag_name-task_name-8vpdausn,
annotations: <omitted>
[2025-06-09T10:15:48.070+0000] {scheduler_job_runner.py:692} INFO - Sending
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) to
KubernetesExecutor with priority 1 and queue default
[2025-06-09T10:15:48.075+0000] {kubernetes_executor.py:352} INFO - Add task
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) with
command ['airflow', 'tasks', 'run', 'dag_name', 'task_name',
'scheduled__2025-06-09T08:00:00+00:00', '--local', '--subdir',
'DAGS_FOLDER/folder_name.py']
[2025-06-09T10:15:48.714+0000] {scheduler_job_runner.py:776} INFO - Received
executor event with state queued for task instance
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1)
[2025-06-09T10:15:49.006+0000] {kubernetes_executor_utils.py:425} INFO -
Creating kubernetes pod for job is TaskInstanceKey(dag_id='dag_name',
task_id='task_name', run_id='scheduled__2025-06-09T08:00:00+00:00',
try_number=1, map_index=-1), with pod name dag_name-task_name-rkrqu4hr,
annotations: <omitted>
[2025-06-09T10:31:50.179+0000] {scheduler_job_runner.py:692} INFO - Sending
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) to
KubernetesExecutor with priority 1 and queue default
[2025-06-09T10:31:50.184+0000] {kubernetes_executor.py:352} INFO - Add task
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1) with
command ['airflow', 'tasks', 'run', 'dag_name', 'task_name',
'scheduled__2025-06-09T08:00:00+00:00', '--local', '--subdir',
'DAGS_FOLDER/folder_name.py']
[2025-06-09T10:31:50.255+0000] {kubernetes_executor_utils.py:425} INFO -
Creating kubernetes pod for job is TaskInstanceKey(dag_id='dag_name',
task_id='task_name', run_id='scheduled__2025-06-09T08:00:00+00:00',
try_number=1, map_index=-1), with pod name dag_name-task_name-rx89exlt,
annotations: <omitted>
[2025-06-09T10:31:51.614+0000] {scheduler_job_runner.py:776} INFO - Received
executor event with state queued for task instance
TaskInstanceKey(dag_id='dag_name', task_id='task_name',
run_id='scheduled__2025-06-09T08:00:00+00:00', try_number=1, map_index=-1)`
```
**Airflow Metadata logs**
TASK 1
```
`airflow=> SELECT * FROM log WHERE dag_id = 'dag_name' AND task_id =
'task_name' AND run_id = 'scheduled__2025-06-09T08:00:00+00:00' LIMIT 10;
id | dttm | dag_id |
task_id | map_index | event
| execution_date | owner |
extra
| owner_display_name | run_id |
try_number
---------+-------------------------------+--------------------+-------------------------+-----------+-------------------------
-------+----------------+-------+---------------------------------------------------------------------------------------------
------+--------------------+--------------------------------------+------------
1802783 | 2025-06-09 10:15:47.409503+00 | dag_name | task_name | -1 |
stuck in queued reschedu
le | | | Task was in queued state for longer than
900.0 seconds; task state will be set back to sched
uled. | | scheduled__2025-06-09T08:00:00+00:00 |
1
1802824 | 2025-06-09 10:31:49.016565+00 | dag_name | task_name | -1 |
stuck in queued reschedu
le | | | Task was in queued state for longer than
900.0 seconds; task state will be set back to sched
uled. | | scheduled__2025-06-09T08:00:00+00:00 |
1
1802844 | 2025-06-09 10:47:50.486896+00 | dag_name | task_name | -1 |
stuck in queued tries ex
ceeded | | | Task was requeued more than 2 times and
will be failed.
| | scheduled__2025-06-09T08:00:00+00:00 |
1`
```
TASK 2
```
`id | dttm | dag_id | task_id
| map_index | event
| execution_date | owner |
extra
| owner_display_name | run_id | try_number
---------+-------------------------------+----------------------+-------------------+-----------+-----------------------------
---+----------------+-------+-------------------------------------------------------------------------------------------------
--+--------------------+--------------------------------------+------------
1699360 | 2025-05-02 14:48:53.464319+00 | dag_name | task_name | -1
| stuck in queued reschedule
| | | Task was in queued state for longer than 900.0
seconds; task state will be set back to scheduled
. | | scheduled__2025-04-30T14:00:00+00:00 | 1
1699381 | 2025-05-02 15:04:55.373482+00 | dag_name | task_name | -1
| stuck in queued reschedule
| | | Task was in queued state for longer than 900.0
seconds; task state will be set back to scheduled
. | | scheduled__2025-04-30T14:00:00+00:00 | 1
1699400 | 2025-05-02 15:21:02.064989+00 | dag_name | task_name | -1
| stuck in queued tries exceed
ed | | | Task was requeued more than 2 times and will
be failed.
| | scheduled__2025-04-30T14:00:00
`
```
Let me know if you need any additional details. Thanks!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
