ahipp13 commented on issue #40957:
URL: https://github.com/apache/airflow/issues/40957#issuecomment-2263146206
Another update for anybody that sees this, tried to look at pod and
scheduler logs combined in our Splunk instance and this is what I found:
For a task that completes and closes normally, the
"kubernetes_executor_utils" will log that it has succeeded, like so:
```
2024-08-01T08:21:01.528695415-05:00 stdout F [2024-08-01T13:21:01.528+0000]
{taskinstance.py:1206} INFO - Marking task as SUCCESS. dag_id=DAG_ID,
task_id=delete_data, run_id=scheduled__2024-08-01T13:10:00+00:00,
execution_date=20240801T131000, start_date=20240801T132059,
end_date=20240801T132101
2024-08-01T08:21:01.587171193-05:00 stdout F [2024-08-01T13:21:01.586+0000]
{local_task_job_runner.py:240} INFO - Task exited with return code 0
2024-08-01T08:21:01.611141571-05:00 stdout F [2024-08-01T13:21:01.610+0000]
{taskinstance.py:3503} INFO - 1 downstream tasks scheduled from follow-on
schedule check
2024-08-01T08:21:01.612627302-05:00 stdout F [2024-08-01T13:21:01.612+0000]
{local_task_job_runner.py:222} INFO - ::endgroup::
2024-08-01T08:21:01.703212706-05:00 stdout F [2024-08-01T13:21:01.701+0000]
{base.py:84} INFO - Using connection ID 'CONN' for task execution.
2024-08-01T08:21:01.924805315-05:00 stdout F [2024-08-01T13:21:01.923+0000]
{get_token_mixin.py:104} INFO - ClientSecretCredential.get_token succeeded
2024-08-01T08:21:02.968331016-05:00 stdout F [2024-08-01T13:21:02.967+0000]
{kubernetes_executor_utils.py:299} INFO - Event: POD_NAME is Running,
annotations: <omitted>
2024-08-01T08:21:04.525216578-05:00 stdout F [2024-08-01T13:21:04.524+0000]
{kubernetes_executor_utils.py:284} INFO - Event: POD_NAME Succeeded,
annotations: <omitted>
```
For a task that stays in a "Running" state, the kubernetes_executor_utils
does not say it succeed and keeps saying it is "Running" even though the task
gets marked as succeeded and returns with an exit code of 0:
```
2024-07-31T23:50:25.538069838-05:00 stdout F [2024-08-01T04:50:25.537+0000]
{base.py:294} INFO - Success criteria met. Exiting.
2024-07-31T23:50:25.553145453-05:00 stdout F [2024-08-01T04:50:25.552+0000]
{taskinstance.py:1206} INFO - Marking task as SUCCESS.
dag_id=secplat_dev_delete_data_alerts, task_id=http_sensor_check,
run_id=scheduled__2024-08-01T04:40:00+00:00, execution_date=20240801T044000,
start_date=20240801T045025, end_date=20240801T045025
2024-07-31T23:50:25.578659388-05:00 stdout F [2024-08-01T04:50:25.577+0000]
{local_task_job_runner.py:240} INFO - Task exited with return code 0
2024-07-31T23:50:25.592799182-05:00 stdout F [2024-08-01T04:50:25.592+0000]
{local_task_job_runner.py:222} INFO - ::endgroup::
2024-07-31T23:50:25.683516859-05:00 stdout F [2024-08-01T04:50:25.681+0000]
{base.py:84} INFO - Using connection ID 'edw_az_conn' for task execution.
2024-07-31T23:50:25.985346073-05:00 stdout F [2024-08-01T04:50:25.984+0000]
{get_token_mixin.py:104} INFO - ClientSecretCredential.get_token succeeded
2024-07-31T23:52:30.827037969-05:00 stdout F [2024-08-01T04:52:30.826+0000]
{kubernetes_executor_utils.py:299} INFO - Event:
secplat-dev-delete-data-alerts-http-sensor-check-179jgg3i is Running,
annotations: <omitted>
(This message goes on and on and on)
```
So my question is, why randomly does the kubernetes_executor_utils not mark
the task as succeeded? @eladkal Any tips on more debugging steps I can take?
Really just don't get why this happens randomly. If I only have 1 scheduler up
I wouldn't think it is a database problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
