doowhtron opened a new issue #13667:
URL: https://github.com/apache/airflow/issues/13667
**Apache Airflow version**: 2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: tencent cloud
- **OS** (e.g. from /etc/os-release): centos7
- **Kernel** (e.g. `uname -a`): 3.10
- **Install tools**:
- **Others**:
**What happened**:
Scheduler dies when I try to restart it. And the logs are as follows:
```
2021-01-14 14:07:44,429} {{scheduler_job.py:1754}} INFO - Resetting orphaned
tasks for active dag runs
{2021-01-14 14:08:14,470} {{scheduler_job.py:1754}} INFO - Resetting
orphaned tasks for active dag runs
{2021-01-14 14:08:16,968} {{scheduler_job.py:1293}} ERROR - Exception when
executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 1275, in _execute
self._run_scheduler_loop()
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 1377, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 1516, in _do_scheduling
self._schedule_dag_run(dag_run,
active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 1629, in _schedule_dag_run
dag = dag_run.dag = self.dagbag.get_dag(dag_run.dag_id, session=session)
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/utils/session.py",
line 62, in wrapper
return func(*args, **kwargs)
File
"/home/app/.pyenv/versions/3.8.1/envs/airflow-py381/lib/python3.8/site-packages/airflow/models/dagbag.py",
line 187, in get_dag
if sd_last_updated_datetime > self.dags_last_fetched[dag_id]:
TypeError: '>' not supported between instances of 'NoneType' and
'datetime.datetime'
{2021-01-14 14:08:17,975} {{process_utils.py:95}} INFO - Sending
Signals.SIGTERM to GPID 53178
{2021-01-14 14:08:18,212} {{process_utils.py:61}} INFO - Process
psutil.Process(pid=58676, status='terminated') (58676) terminated with exit
code None
{2021-01-14 14:08:18,295} {{process_utils.py:201}} INFO - Waiting up to 5
seconds for processes to exit...
{2021-01-14 14:08:18,345} {{process_utils.py:61}} INFO - Process
psutil.Process(pid=53178, status='terminated') (53178) terminated with exit
code 0
{2021-01-14 14:08:18,345} {{process_utils.py:61}} INFO - Process
psutil.Process(pid=58677, status='terminated') (58677) terminated with exit
code None
{2021-01-14 14:08:18,346} {{process_utils.py:61}} INFO - Process
psutil.Process(pid=58678, status='terminated') (58678) terminated with exit
code None
{2021-01-14 14:08:18,346} {{process_utils.py:61}} INFO - Process
psutil.Process(pid=58708, status='terminated') (58708) terminated with exit
code None
{2021-01-14 14:08:18,346} {{scheduler_job.py:1296}} INFO - Exited execute
loop
```
**What you expected to happen**:
Schdeduler should not die.
**How to reproduce it**:
I don't know how to reproduce it
**Anything else we need to know**:
I just upgrade airflow from 1.10.14. Now I try to fix it temporarily by
catching the exception in scheduler_job.py
```python
for dag_run in dag_runs:
try:
self._schedule_dag_run(dag_run,
active_runs_by_dag_id.get(dag_run.dag_id, set()), session)
except Exception as e:
self.log.exception(e)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
