arjunanan6 commented on issue #33402:
URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678787624

   This new set of logs shows where the Deletion process suddenly stops:
   
   ```
   [2023-08-15T11:01:58.842+0000] {kubernetes_executor.py:764} INFO - Deleted 
pod: TaskInstanceKey(dag_id='WINRM_TESTER', task_id='WINRM_ECHO', 
run_id='manual__2023-08-15T11:01:26.967568+00:00', try_number=1, map_index=-1) 
in namespace airflow-test-ns
   [2023-08-15T11:01:58.843+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod winrm-tester-winrm-echo-j94naf9j - event for this pod 
already sent to executor
   [2023-08-15T11:01:59.088+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state success for task instance 
TaskInstanceKey(dag_id='WINRM_TESTER', task_id='WINRM_ECHO', 
run_id='manual__2023-08-15T11:01:26.967568+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:01:59.335+0000] {scheduler_job_runner.py:713} INFO - 
TaskInstance Finished: dag_id=WINRM_TESTER, task_id=WINRM_ECHO, 
run_id=manual__2023-08-15T11:01:26.967568+00:00, map_index=-1, 
run_start_date=2023-08-15 11:01:44.008014+00:00, run_end_date=2023-08-15 
11:01:52.020878+00:00, run_duration=8.012864, state=success, 
executor_state=success, try_number=1, max_tries=0, job_id=219355, 
pool=default_pool, queue=default, priority_weight=1, operator=PsrpOperator, 
queued_dttm=2023-08-15 11:01:31.223329+00:00, queued_by_job_id=219351, pid=15
   [2023-08-15T11:02:15.279+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:02:34.784+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:03:35.602+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:04:37.343+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:05:02.370+0000] {scheduler_job_runner.py:1294} INFO - DAG 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H is at (or above) max_active_runs (1 of 1), 
not creating any more runs
   [2023-08-15T11:05:02.691+0000] {scheduler_job_runner.py:1294} INFO - DAG 
SAP_WMD_SEND_MAIL_TO_RELEASER is at (or above) max_active_runs (1 of 1), not 
creating any more runs
   [2023-08-15T11:05:02.989+0000] {scheduler_job_runner.py:1294} INFO - DAG 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H is at (or above) max_active_runs (1 of 1), not 
creating any more runs
   [2023-08-15T11:05:05.221+0000] {scheduler_job_runner.py:411} INFO - 3 tasks 
up for execution:
        <TaskInstance: 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H.SAP_WMD_AGWI_RSRVD_TIMEOUT_H 
scheduled__2023-08-15T10:05:00+00:00 [scheduled]>
        <TaskInstance: 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H.IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H 
scheduled__2023-08-15T10:05:00+00:00 [scheduled]>
        <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T10:50:00+00:00 [scheduled]>
   [2023-08-15T11:05:05.221+0000] {scheduler_job_runner.py:476} INFO - DAG 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H has 0/16 running and queued tasks
   [2023-08-15T11:05:05.221+0000] {scheduler_job_runner.py:476} INFO - DAG 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H has 0/16 running and queued tasks
   [2023-08-15T11:05:05.221+0000] {scheduler_job_runner.py:476} INFO - DAG 
SAP_WMD_SEND_MAIL_TO_RELEASER has 0/16 running and queued tasks
   [2023-08-15T11:05:05.222+0000] {scheduler_job_runner.py:587} INFO - Setting 
the following tasks to queued state:
        <TaskInstance: 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H.SAP_WMD_AGWI_RSRVD_TIMEOUT_H 
scheduled__2023-08-15T10:05:00+00:00 [scheduled]>
        <TaskInstance: 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H.IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H 
scheduled__2023-08-15T10:05:00+00:00 [scheduled]>
        <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T10:50:00+00:00 [scheduled]>
   [2023-08-15T11:05:05.296+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:05:05.296+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
'SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 'scheduled__2023-08-15T10:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_AGWI_RSRVD_TIMEOUT_H.py']
   [2023-08-15T11:05:05.296+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:05:05.297+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
'IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 'scheduled__2023-08-15T10:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H.py']
   [2023-08-15T11:05:05.297+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:05:05.297+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'SAP_WMD_SEND_MAIL_TO_RELEASER', 
'SAP_WMD_SEND_MAIL_TO_RELEASER', 'scheduled__2023-08-15T10:50:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_SEND_MAIL_TO_RELEASER.py']
   [2023-08-15T11:05:05.373+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
'SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 'scheduled__2023-08-15T10:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_AGWI_RSRVD_TIMEOUT_H.py']
   [2023-08-15T11:05:05.374+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
'IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 'scheduled__2023-08-15T10:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H.py']
   [2023-08-15T11:05:05.374+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'SAP_WMD_SEND_MAIL_TO_RELEASER', 
'SAP_WMD_SEND_MAIL_TO_RELEASER', 'scheduled__2023-08-15T10:50:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_SEND_MAIL_TO_RELEASER.py']
   [2023-08-15T11:05:05.407+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1), 
with pod name sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph
   [2023-08-15T11:05:05.543+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1), 
with pod name 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90
   [2023-08-15T11:05:05.660+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1), 
with pod name 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71
   [2023-08-15T11:05:05.758+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:05:05.759+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:05:05.759+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:05:06.018+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H.IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H 
scheduled__2023-08-15T10:05:00+00:00 [queued]> to 219351
   [2023-08-15T11:05:06.018+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H.SAP_WMD_AGWI_RSRVD_TIMEOUT_H 
scheduled__2023-08-15T10:05:00+00:00 [queued]> to 219351
   [2023-08-15T11:05:06.019+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T10:50:00+00:00 [queued]> to 219351
   [2023-08-15T11:05:07.201+0000] {kubernetes_executor.py:252} INFO - Event: 
sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph is Running
   [2023-08-15T11:05:07.543+0000] {kubernetes_executor.py:252} INFO - Event: 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90 
is Running
   [2023-08-15T11:05:08.241+0000] {kubernetes_executor.py:252} INFO - Event: 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71 is Running
   [2023-08-15T11:05:29.371+0000] {dagrun.py:630} INFO - Marking run <DagRun 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H @ 2023-08-15 10:05:00+00:00: 
scheduled__2023-08-15T10:05:00+00:00, state:running, queued_at: 2023-08-15 
11:05:02.752626+00:00. externally triggered: False> successful
   [2023-08-15T11:05:29.371+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=SAP_WMD_AGWI_RSRVD_TIMEOUT_H, execution_date=2023-08-15 10:05:00+00:00, 
run_id=scheduled__2023-08-15T10:05:00+00:00, run_start_date=2023-08-15 
11:05:03.342966+00:00, run_end_date=2023-08-15 11:05:29.371496+00:00, 
run_duration=26.02853, state=success, external_trigger=False, 
run_type=scheduled, data_interval_start=2023-08-15 10:05:00+00:00, 
data_interval_end=2023-08-15 11:05:00+00:00, 
dag_hash=4174ce1675c90b2a3d988a0732ef8eb0
   [2023-08-15T11:05:29.513+0000] {dag.py:3504} INFO - Setting next_dagrun for 
SAP_WMD_AGWI_RSRVD_TIMEOUT_H to 2023-08-15T11:05:00+00:00, 
run_after=2023-08-15T12:05:00+00:00
   [2023-08-15T11:05:31.313+0000] {kubernetes_executor.py:252} INFO - Event: 
sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph is Running
   [2023-08-15T11:05:32.506+0000] {kubernetes_executor.py:243} INFO - Event: 
sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph Succeeded
   [2023-08-15T11:05:35.470+0000] {kubernetes_executor.py:679} INFO - Changing 
state of (TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1), 
None, 'sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph', 
'airflow-test-ns', '359283135') to None
   [2023-08-15T11:05:35.504+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph - event for 
this pod already sent to executor
   [2023-08-15T11:05:35.516+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
sap-wmd-agwi-rsrvd-timeout-h-sap-wmd-agwi-rsrvd-timeout-h-uq0ks5ph - event for 
this pod already sent to executor
   [2023-08-15T11:05:35.518+0000] {kubernetes_executor.py:764} INFO - Deleted 
pod: TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) in 
namespace airflow-test-ns
   [2023-08-15T11:05:35.757+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state success for task instance 
TaskInstanceKey(dag_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
task_id='SAP_WMD_AGWI_RSRVD_TIMEOUT_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:05:36.005+0000] {scheduler_job_runner.py:713} INFO - 
TaskInstance Finished: dag_id=SAP_WMD_AGWI_RSRVD_TIMEOUT_H, 
task_id=SAP_WMD_AGWI_RSRVD_TIMEOUT_H, 
run_id=scheduled__2023-08-15T10:05:00+00:00, map_index=-1, 
run_start_date=2023-08-15 11:05:19.585526+00:00, run_end_date=2023-08-15 
11:05:27.933818+00:00, run_duration=8.348292, state=success, 
executor_state=success, try_number=1, max_tries=0, job_id=219356, 
pool=default_pool, queue=default, priority_weight=1, operator=SapOperator, 
queued_dttm=2023-08-15 11:05:05.223443+00:00, queued_by_job_id=219351, pid=15
   [2023-08-15T11:05:39.367+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:06:03.414+0000] {kubernetes_executor.py:252} INFO - Event: 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71 is Running
   [2023-08-15T11:06:03.706+0000] {dagrun.py:630} INFO - Marking run <DagRun 
SAP_WMD_SEND_MAIL_TO_RELEASER @ 2023-08-15 10:50:00+00:00: 
scheduled__2023-08-15T10:50:00+00:00, state:running, queued_at: 2023-08-15 
11:05:02.434440+00:00. externally triggered: False> successful
   [2023-08-15T11:06:03.706+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=SAP_WMD_SEND_MAIL_TO_RELEASER, execution_date=2023-08-15 10:50:00+00:00, 
run_id=scheduled__2023-08-15T10:50:00+00:00, run_start_date=2023-08-15 
11:05:03.400414+00:00, run_end_date=2023-08-15 11:06:03.706379+00:00, 
run_duration=60.305965, state=success, external_trigger=False, 
run_type=scheduled, data_interval_start=2023-08-15 10:50:00+00:00, 
data_interval_end=2023-08-15 11:05:00+00:00, 
dag_hash=7d510821b53e85efc31b025b52cf7fda
   [2023-08-15T11:06:03.827+0000] {dag.py:3504} INFO - Setting next_dagrun for 
SAP_WMD_SEND_MAIL_TO_RELEASER to 2023-08-15T11:05:00+00:00, 
run_after=2023-08-15T11:20:00+00:00
   [2023-08-15T11:06:05.431+0000] {kubernetes_executor.py:243} INFO - Event: 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71 Succeeded
   [2023-08-15T11:06:07.965+0000] {kubernetes_executor.py:679} INFO - Changing 
state of (TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1), 
None, 'sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71', 
'airflow-test-ns', '359283445') to None
   [2023-08-15T11:06:08.003+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71 - event 
for this pod already sent to executor
   [2023-08-15T11:06:08.008+0000] {kubernetes_executor.py:764} INFO - Deleted 
pod: TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1) in 
namespace airflow-test-ns
   [2023-08-15T11:06:08.014+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-qax7hn71 - event 
for this pod already sent to executor
   [2023-08-15T11:06:08.260+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state success for task instance 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T10:50:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:06:08.506+0000] {scheduler_job_runner.py:713} INFO - 
TaskInstance Finished: dag_id=SAP_WMD_SEND_MAIL_TO_RELEASER, 
task_id=SAP_WMD_SEND_MAIL_TO_RELEASER, 
run_id=scheduled__2023-08-15T10:50:00+00:00, map_index=-1, 
run_start_date=2023-08-15 11:05:20.526361+00:00, run_end_date=2023-08-15 
11:06:00.451028+00:00, run_duration=39.924667, state=success, 
executor_state=success, try_number=1, max_tries=0, job_id=219358, 
pool=default_pool, queue=default, priority_weight=1, operator=SapOperator, 
queued_dttm=2023-08-15 11:05:05.223443+00:00, queued_by_job_id=219351, pid=15
   [2023-08-15T11:06:41.106+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:07:03.187+0000] {dagrun.py:630} INFO - Marking run <DagRun 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H @ 2023-08-15 10:05:00+00:00: 
scheduled__2023-08-15T10:05:00+00:00, state:running, queued_at: 2023-08-15 
11:05:02.186700+00:00. externally triggered: False> successful
   [2023-08-15T11:07:03.187+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H, execution_date=2023-08-15 
10:05:00+00:00, run_id=scheduled__2023-08-15T10:05:00+00:00, 
run_start_date=2023-08-15 11:05:03.342390+00:00, run_end_date=2023-08-15 
11:07:03.187387+00:00, run_duration=119.844997, state=success, 
external_trigger=False, run_type=scheduled, data_interval_start=2023-08-15 
10:05:00+00:00, data_interval_end=2023-08-15 11:05:00+00:00, 
dag_hash=594936f0f47ab003a8c5dbe29b157fe6
   [2023-08-15T11:07:03.311+0000] {dag.py:3504} INFO - Setting next_dagrun for 
IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H to 2023-08-15T11:05:00+00:00, 
run_after=2023-08-15T12:05:00+00:00
   [2023-08-15T11:07:03.608+0000] {kubernetes_executor.py:252} INFO - Event: 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90 
is Running
   [2023-08-15T11:07:05.623+0000] {kubernetes_executor.py:243} INFO - Event: 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90 
Succeeded
   [2023-08-15T11:07:06.672+0000] {kubernetes_executor.py:679} INFO - Changing 
state of (TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1), 
None, 
'it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90',
 'airflow-test-ns', '359283970') to None
   [2023-08-15T11:07:06.722+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90 
- event for this pod already sent to executor
   [2023-08-15T11:07:06.731+0000] {kubernetes_executor.py:764} INFO - Deleted 
pod: TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1) in 
namespace airflow-test-ns
   [2023-08-15T11:07:06.734+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod 
it-erpbut-grac-action-usage-sync-h-it-erpbut-grac-action-usage-sync-h-z6d4gx90 
- event for this pod already sent to executor
   [2023-08-15T11:07:06.962+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state success for task instance 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
task_id='IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H', 
run_id='scheduled__2023-08-15T10:05:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:07:07.194+0000] {scheduler_job_runner.py:713} INFO - 
TaskInstance Finished: dag_id=IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H, 
task_id=IT_ERPBUT_GRAC_ACTION_USAGE_SYNC_H, 
run_id=scheduled__2023-08-15T10:05:00+00:00, map_index=-1, 
run_start_date=2023-08-15 11:05:19.698207+00:00, run_end_date=2023-08-15 
11:07:01.138774+00:00, run_duration=101.440567, state=success, 
executor_state=success, try_number=1, max_tries=0, job_id=219357, 
pool=default_pool, queue=default, priority_weight=1, operator=SapOperator, 
queued_dttm=2023-08-15 11:05:05.223443+00:00, queued_by_job_id=219351, pid=15
   [2023-08-15T11:07:17.009+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:07:43.683+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:08:44.531+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:09:48.944+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:10:49.962+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:11:50.787+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:12:18.352+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:12:51.639+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:13:54.336+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:14:55.611+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:15:00.820+0000] {scheduler_job_runner.py:1294} INFO - DAG 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H is at (or above) max_active_runs (1 of 1), 
not creating any more runs
   [2023-08-15T11:15:02.411+0000] {scheduler_job_runner.py:411} INFO - 1 tasks 
up for execution:
        <TaskInstance: 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H.IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H 
scheduled__2023-08-15T10:15:00+00:00 [scheduled]>
   [2023-08-15T11:15:02.412+0000] {scheduler_job_runner.py:476} INFO - DAG 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H has 0/16 running and queued tasks
   [2023-08-15T11:15:02.412+0000] {scheduler_job_runner.py:587} INFO - Setting 
the following tasks to queued state:
        <TaskInstance: 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H.IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H 
scheduled__2023-08-15T10:15:00+00:00 [scheduled]>
   [2023-08-15T11:15:02.468+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
task_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
run_id='scheduled__2023-08-15T10:15:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:15:02.468+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
'IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 'scheduled__2023-08-15T10:15:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H.py']
   [2023-08-15T11:15:02.527+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
task_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
run_id='scheduled__2023-08-15T10:15:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
'IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 'scheduled__2023-08-15T10:15:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H.py']
   [2023-08-15T11:15:02.558+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
task_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
run_id='scheduled__2023-08-15T10:15:00+00:00', try_number=1, map_index=-1), 
with pod name 
it-erpbut-grac-batch-risk-analysis-h-it-erpbut-grac-batch-risk-analysis-mrjn43ae
   [2023-08-15T11:15:02.650+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
task_id='IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H', 
run_id='scheduled__2023-08-15T10:15:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:15:02.867+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H.IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H 
scheduled__2023-08-15T10:15:00+00:00 [queued]> to 219351
   [2023-08-15T11:15:57.951+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:16:24.013+0000] {dagrun.py:609} ERROR - Marking run <DagRun 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H @ 2023-08-15 10:15:00+00:00: 
scheduled__2023-08-15T10:15:00+00:00, state:running, queued_at: 2023-08-15 
11:15:00.626376+00:00. externally triggered: False> failed
   [2023-08-15T11:16:24.013+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H, execution_date=2023-08-15 
10:15:00+00:00, run_id=scheduled__2023-08-15T10:15:00+00:00, 
run_start_date=2023-08-15 11:15:01.168622+00:00, run_end_date=2023-08-15 
11:16:24.013704+00:00, run_duration=82.845082, state=failed, 
external_trigger=False, run_type=scheduled, data_interval_start=2023-08-15 
10:15:00+00:00, data_interval_end=2023-08-15 11:15:00+00:00, 
dag_hash=a8557dbe0e3a29365e4ac7c7bede076b
   [2023-08-15T11:16:24.122+0000] {dag.py:3504} INFO - Setting next_dagrun for 
IT_ERPBUT_GRAC_BATCH_RISK_ANALYSIS_H to 2023-08-15T11:15:00+00:00, 
run_after=2023-08-15T12:15:00+00:00
   [2023-08-15T11:17:00.031+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:17:19.392+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:18:01.554+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:19:02.990+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:20:00.878+0000] {scheduler_job_runner.py:1294} INFO - DAG 
SAP_WMD_SEND_MAIL_TO_RELEASER is at (or above) max_active_runs (1 of 1), not 
creating any more runs
   [2023-08-15T11:20:02.373+0000] {scheduler_job_runner.py:411} INFO - 1 tasks 
up for execution:
        <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T11:05:00+00:00 [scheduled]>
   [2023-08-15T11:20:02.373+0000] {scheduler_job_runner.py:476} INFO - DAG 
SAP_WMD_SEND_MAIL_TO_RELEASER has 0/16 running and queued tasks
   [2023-08-15T11:20:02.373+0000] {scheduler_job_runner.py:587} INFO - Setting 
the following tasks to queued state:
        <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T11:05:00+00:00 [scheduled]>
   [2023-08-15T11:20:02.427+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T11:05:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:20:02.427+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'SAP_WMD_SEND_MAIL_TO_RELEASER', 
'SAP_WMD_SEND_MAIL_TO_RELEASER', 'scheduled__2023-08-15T11:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_SEND_MAIL_TO_RELEASER.py']
   [2023-08-15T11:20:02.488+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T11:05:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'SAP_WMD_SEND_MAIL_TO_RELEASER', 
'SAP_WMD_SEND_MAIL_TO_RELEASER', 'scheduled__2023-08-15T11:05:00+00:00', 
'--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_WMD/SAP_WMD_SEND_MAIL_TO_RELEASER.py']
   [2023-08-15T11:20:02.519+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T11:05:00+00:00', try_number=1, map_index=-1), 
with pod name 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-aw2m0s5a
   [2023-08-15T11:20:02.626+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
task_id='SAP_WMD_SEND_MAIL_TO_RELEASER', 
run_id='scheduled__2023-08-15T11:05:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:20:02.838+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
SAP_WMD_SEND_MAIL_TO_RELEASER.SAP_WMD_SEND_MAIL_TO_RELEASER 
scheduled__2023-08-15T11:05:00+00:00 [queued]> to 219351
   [2023-08-15T11:20:04.795+0000] {kubernetes_executor.py:552} INFO - Found 1 
queued task instances
   [2023-08-15T11:20:56.525+0000] {dagrun.py:630} INFO - Marking run <DagRun 
SAP_WMD_SEND_MAIL_TO_RELEASER @ 2023-08-15 11:05:00+00:00: 
scheduled__2023-08-15T11:05:00+00:00, state:running, queued_at: 2023-08-15 
11:20:00.715535+00:00. externally triggered: False> successful
   [2023-08-15T11:20:56.525+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=SAP_WMD_SEND_MAIL_TO_RELEASER, execution_date=2023-08-15 11:05:00+00:00, 
run_id=scheduled__2023-08-15T11:05:00+00:00, run_start_date=2023-08-15 
11:20:01.244566+00:00, run_end_date=2023-08-15 11:20:56.525351+00:00, 
run_duration=55.280785, state=success, external_trigger=False, 
run_type=scheduled, data_interval_start=2023-08-15 11:05:00+00:00, 
data_interval_end=2023-08-15 11:20:00+00:00, 
dag_hash=7d510821b53e85efc31b025b52cf7fda
   [2023-08-15T11:20:56.690+0000] {dag.py:3504} INFO - Setting next_dagrun for 
SAP_WMD_SEND_MAIL_TO_RELEASER to 2023-08-15T11:20:00+00:00, 
run_after=2023-08-15T11:35:00+00:00
   [2023-08-15T11:21:05.216+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:22:06.765+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:22:20.776+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:23:08.407+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:24:10.426+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:25:02.342+0000] {scheduler_job_runner.py:1294} INFO - DAG 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H is at (or above) max_active_runs (1 of 
1), not creating any more runs
   [2023-08-15T11:25:03.978+0000] {scheduler_job_runner.py:411} INFO - 1 tasks 
up for execution:
        <TaskInstance: 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H.IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H 
scheduled__2023-08-15T10:25:00+00:00 [scheduled]>
   [2023-08-15T11:25:03.978+0000] {scheduler_job_runner.py:476} INFO - DAG 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H has 0/16 running and queued tasks
   [2023-08-15T11:25:03.978+0000] {scheduler_job_runner.py:587} INFO - Setting 
the following tasks to queued state:
        <TaskInstance: 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H.IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H 
scheduled__2023-08-15T10:25:00+00:00 [scheduled]>
   [2023-08-15T11:25:04.043+0000] {scheduler_job_runner.py:625} INFO - Sending 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
task_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
run_id='scheduled__2023-08-15T10:25:00+00:00', try_number=1, map_index=-1) to 
executor with priority 1 and queue default
   [2023-08-15T11:25:04.043+0000] {base_executor.py:147} INFO - Adding to 
queue: ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
'IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
'scheduled__2023-08-15T10:25:00+00:00', '--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H.py']
   [2023-08-15T11:25:04.113+0000] {kubernetes_executor.py:637} INFO - Add task 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
task_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
run_id='scheduled__2023-08-15T10:25:00+00:00', try_number=1, map_index=-1) with 
command ['airflow', 'tasks', 'run', 'IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
'IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
'scheduled__2023-08-15T10:25:00+00:00', '--local', '--subdir', 
'DAGS_FOLDER/airflow-dags-sap/SAP_ITGCP/test_flow/IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H.py']
   [2023-08-15T11:25:04.145+0000] {kubernetes_executor.py:379} INFO - Creating 
kubernetes pod for job is 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
task_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
run_id='scheduled__2023-08-15T10:25:00+00:00', try_number=1, map_index=-1), 
with pod name 
it-erpbut-grac-repository-object-sync-h-it-erpbut-grac-repository-objec-45u1nfx2
   [2023-08-15T11:25:04.259+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
task_id='IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H', 
run_id='scheduled__2023-08-15T10:25:00+00:00', try_number=1, map_index=-1)
   [2023-08-15T11:25:04.538+0000] {scheduler_job_runner.py:703} INFO - Setting 
external_id for <TaskInstance: 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H.IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H 
scheduled__2023-08-15T10:25:00+00:00 [queued]> to 219351
   [2023-08-15T11:25:10.965+0000] {kubernetes_executor.py:552} INFO - Found 1 
queued task instances
   [2023-08-15T11:26:11.964+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:27:12.781+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   [2023-08-15T11:27:22.697+0000] {scheduler_job_runner.py:1553} INFO - 
Resetting orphaned tasks for active dag runs
   [2023-08-15T11:28:02.569+0000] {dagrun.py:630} INFO - Marking run <DagRun 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H @ 2023-08-15 10:25:00+00:00: 
scheduled__2023-08-15T10:25:00+00:00, state:running, queued_at: 2023-08-15 
11:25:02.148542+00:00. externally triggered: False> successful
   [2023-08-15T11:28:02.569+0000] {dagrun.py:681} INFO - DagRun Finished: 
dag_id=IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H, execution_date=2023-08-15 
10:25:00+00:00, run_id=scheduled__2023-08-15T10:25:00+00:00, 
run_start_date=2023-08-15 11:25:02.722844+00:00, run_end_date=2023-08-15 
11:28:02.569315+00:00, run_duration=179.846471, state=success, 
external_trigger=False, run_type=scheduled, data_interval_start=2023-08-15 
10:25:00+00:00, data_interval_end=2023-08-15 11:25:00+00:00, 
dag_hash=975811dc8090d6735ee3e36e00c0f79c
   [2023-08-15T11:28:02.674+0000] {dag.py:3504} INFO - Setting next_dagrun for 
IT_ERPBUT_GRAC_REPOSITORY_OBJECT_SYNC_H to 2023-08-15T11:25:00+00:00, 
run_after=2023-08-15T12:25:00+00:00
   [2023-08-15T11:28:15.003+0000] {kubernetes_executor.py:552} INFO - Found 0 
queued task instances
   ```
   
   
   Last instance of a successful worker pod deletion is at 
2023-08-15T11:07:06.731. Then, `kubernetes_executor` no longer checks events 
for the pod state (Running, Succeeded, etc.), and thus the worker pods don't 
get deleted (at least that is my observation). I cannot see any errors either. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to