tcchong opened a new issue #20944:
URL: https://github.com/apache/airflow/issues/20944
### Apache Airflow version
2.2.2
### What happened
I tried to run a backfilling job with k8s executor, it's created the pod and
running the job well.
When the pod status mark as `Completed` in k8s, the status of task get stuck
in `scheduled` state without any updates.
### What you expected to happen
I would expect the state will change once the k8s pod had complete the job.
This is the scheduler and executor logs:
```
[2022-01-19 08:40:14,670] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:40:24,973] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:40:35,358] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:40:45,655] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:40:55,828] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:05,981] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:16,138] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:26,299] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:36,587] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:46,939] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:41:51,823] {scheduler_job.py:1114} INFO - Resetting orphaned
tasks for active dag runs
[2022-01-19 08:41:51,857] {kubernetes_executor.py:730} INFO - Attempting to
adopt pod echostart.9babbb429d7f43a98ff44f2976b59343
[2022-01-19 08:41:51,884] {kubernetes_executor.py:730} INFO - Attempting to
adopt pod echostart.d902f94926784e27afcef857f90e7411
[2022-01-19 08:41:51,887] {kubernetes_executor.py:147} INFO - Event:
echostart.9babbb429d7f43a98ff44f2976b59343 had an event of type ADDED
[2022-01-19 08:41:51,888] {kubernetes_executor.py:206} INFO - Event:
echostart.9babbb429d7f43a98ff44f2976b59343 Succeeded
[2022-01-19 08:41:51,905] {kubernetes_executor.py:147} INFO - Event:
echostart.d902f94926784e27afcef857f90e7411 had an event of type ADDED
[2022-01-19 08:41:51,905] {kubernetes_executor.py:206} INFO - Event:
echostart.d902f94926784e27afcef857f90e7411 Succeeded
[2022-01-19 08:41:52,019] {kubernetes_executor.py:374} INFO - Attempting to
finish pod; pod_id: echostart.9babbb429d7f43a98ff44f2976b59343; state: None;
annotations: {'dag_id': 'echo', 'task_id': 'start', 'execution_date': None,
'run_id': 'backfill__2022-01-17T00:00:00+00:00', 'try_number': '1'}
[2022-01-19 08:41:52,020] {kubernetes_executor.py:374} INFO - Attempting to
finish pod; pod_id: echostart.d902f94926784e27afcef857f90e7411; state: None;
annotations: {'dag_id': 'echo', 'task_id': 'start', 'execution_date': None,
'run_id': 'scheduled__2022-01-18T00:00:00+00:00', 'try_number': '1'}
[2022-01-19 08:41:52,021] {kubernetes_executor.py:576} INFO - Changing state
of (TaskInstanceKey(dag_id='echo', task_id='start',
run_id='backfill__2022-01-17T00:00:00+00:00', try_number=1), None,
'echostart.9babbb429d7f43a98ff44f2976b59343', 'airflow-staging', '352215076')
to None
[2022-01-19 08:41:52,022] {kubernetes_executor.py:576} INFO - Changing state
of (TaskInstanceKey(dag_id='echo', task_id='start',
run_id='scheduled__2022-01-18T00:00:00+00:00', try_number=1), None,
'echostart.d902f94926784e27afcef857f90e7411', 'airflow-staging', '352215077')
to None
[2022-01-19 08:41:52,023] {scheduler_job.py:504} INFO - Executor reports
execution of echo.start run_id=backfill__2022-01-17T00:00:00+00:00 exited with
status None for try_number 1
[2022-01-19 08:41:52,024] {scheduler_job.py:504} INFO - Executor reports
execution of echo.start run_id=scheduled__2022-01-18T00:00:00+00:00 exited with
status None for try_number 1
[2022-01-19 08:41:57,107] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:07,279] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:17,419] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:27,574] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:37,743] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:48,097] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
[2022-01-19 08:42:58,292] {kubernetes_executor.py:454} INFO - Found 0 queued
task instances
```
Pod status:
```
NAME READY STATUS RESTARTS
AGE
echostart.9babbb429d7f43a98ff44f2976b59343 0/1 Completed 0
3m55s
echostart.d902f94926784e27afcef857f90e7411 0/1 Completed 0
3m55s
```
Airflow Web:
### How to reproduce
1. Create a DAG as below
```python
from datetime import timedelta
from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
default_args = {
'owner': 'echo',
'depends_on_past': False,
'start_date': days_ago(2),
'email': [],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=1),
'on_failure_callback': failure_alert,
'timeout': 60 * 60
}
with DAG('echo', default_args=default_args, schedule_interval='@daily') as
dag:
start = DummyOperator(task_id='start')
hello = BashOperator(
task_id='hello',
bash_command='echo hello',
)
end = DummyOperator(task_id='end')
start >> hello >> end
```
2. Run backfill command
```
$ airflow dags backfill echo -s 20220117
```
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
