svncsvr opened a new issue, #23948:
URL: https://github.com/apache/airflow/issues/23948
### Apache Airflow version
2.2.3
### What happened
First of all, It is a very strange error and happened all of a sudden when
running some tasks within a DAG which was switched off
So basically within this DAG, we had a sensor running and a few downstream
after that in a clean state. After those were in a clean state another sensor
was also running. At this point, the scheduler started crashing
We are using CeleryExecutor and running Airflow v2.2.3
### What you think should happen instead
_No response_
### How to reproduce
I don't really know but here is the error stack trace
=1653590702662,
https://airflow.apache.org/docs/docker-stack/build.html
1653590716300,BACKEND=redis
1653590716301,DB_HOST=airflow-de-preprod.ns1d7y.0001.euc1.cache.amazonaws.com
1653590716301,DB_PORT=6379
1653590719192, ____________ _____________
1653590719192, ____ |__( )_________ __/__ /________ __
1653590719192,____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
1653590719192,___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
1653590719192, _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
1653590719192,"[[34m2022-05-26 18:45:19,191[0m]
{[34mscheduler_job.py:[0m596} INFO[0m - Starting the scheduler[0m"
1653590719192,"[[34m2022-05-26 18:45:19,192[0m]
{[34mscheduler_job.py:[0m601} INFO[0m - Processing each file at most -1
times[0m"
1653590719417,"[[34m2022-05-26 18:45:19,412[0m] {[34mmanager.py:[0m163}
INFO[0m - Launched DagFileProcessorManager with pid: 38[0m"
1653590719417,"[[34m2022-05-26 18:45:19,415[0m]
{[34mscheduler_job.py:[0m1114} INFO[0m - Resetting orphaned tasks for active
dag runs[0m"
1653590719421,"[[34m2022-05-26 18:45:19,420[0m] {[34msettings.py:[0m52}
INFO[0m - Configured default timezone Timezone('Europe/Berlin')[0m"
1653590719423,"[[34m2022-05-26 18:45:19,423[0m] {[34msettings.py:[0m471}
INFO[0m - Loaded airflow_local_settings from
/opt/airflow/config/airflow_local_settings.py .[0m"
1653590719563,"[[34m2022-05-26 18:45:19,559[0m]
{[34mscheduler_job.py:[0m644} ERROR[0m - Exception when executing
SchedulerJob._run_scheduler_loop[0m"
1653590719563,Traceback (most recent call last):
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 628, in _execute"
1653590719563, self._run_scheduler_loop()
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 681, in _run_scheduler_loop"
1653590719563, self.adopt_or_reset_orphaned_tasks()
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py"",
line 70, in wrapper"
1653590719563," return func(*args, session=session, **kwargs)"
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 1117, in adopt_or_reset_orphaned_tasks"
1653590719563, for attempt in run_with_db_retries(logger=self.log):
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line
382, in __iter__"
1653590719563, do = self.iter(retry_state=retry_state)
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line
349, in iter"
1653590719563, return fut.result()
1653590719563," File
""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 428, in result"
1653590719563, return self.__get_result()
1653590719563," File
""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 384, in
__get_result"
1653590719563, raise self._exception
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 1162, in adopt_or_reset_orphaned_tasks"
1653590719563, to_reset =
self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
1653590719563," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py"",
line 485, in try_adopt_task_instances"
1653590719563, self.adopted_task_timeouts[ti.key] = ti.queued_dttm +
self.task_adoption_timeout
1653590719563,TypeError: unsupported operand type(s) for +: 'NoneType' and
'datetime.timedelta'
1653590720596,"[[34m2022-05-26 18:45:20,596[0m]
{[34mprocess_utils.py:[0m124} INFO[0m - Sending Signals.SIGTERM to group 38.
PIDs of all processes in the group: [39, 40, 38][0m"
1653590720597,"[[34m2022-05-26 18:45:20,596[0m]
{[34mprocess_utils.py:[0m75} INFO[0m - Sending the signal Signals.SIGTERM to
group 38[0m"
1653590721273,"[2022-05-26 18:45:21,272] {process_utils.py:237} INFO -
Waiting up to 5 seconds for processes to exit..."
1653590721276,"[[34m2022-05-26 18:45:21,276[0m]
{[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=40,
status='terminated', started='18:45:19') (40) terminated with exit code
None[0m"
1653590721280,"[[34m2022-05-26 18:45:21,280[0m]
{[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=38,
status='terminated', exitcode=0, started='18:45:18') (38) terminated with exit
code 0[0m"
1653590721281,"[[34m2022-05-26 18:45:21,280[0m]
{[34mprocess_utils.py:[0m70} INFO[0m - Process psutil.Process(pid=39,
status='terminated', started='18:45:19') (39) terminated with exit code
None[0m"
1653590721281,"[[34m2022-05-26 18:45:21,281[0m]
{[34mscheduler_job.py:[0m655} INFO[0m - Exited execute loop[0m"
1653590721294,Traceback (most recent call last):
1653590721294," File ""/home/airflow/.local/bin/airflow"", line 8, in
<module>"
1653590721294, sys.exit(main())
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py"", line
48, in main"
1653590721294, args.func(args)
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py"",
line 48, in command"
1653590721294," return func(*args, **kwargs)"
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py"", line
92, in wrapper"
1653590721294," return f(*args, **kwargs)"
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py"",
line 75, in scheduler"
1653590721294, _run_scheduler_job(args=args)
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py"",
line 46, in _run_scheduler_job"
1653590721294, job.run()
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py"",
line 245, in run"
1653590721294, self._execute()
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 628, in _execute"
1653590721294, self._run_scheduler_loop()
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 681, in _run_scheduler_loop"
1653590721294, self.adopt_or_reset_orphaned_tasks()
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py"",
line 70, in wrapper"
1653590721294," return func(*args, session=session, **kwargs)"
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 1117, in adopt_or_reset_orphaned_tasks"
1653590721294, for attempt in run_with_db_retries(logger=self.log):
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line
382, in __iter__"
1653590721294, do = self.iter(retry_state=retry_state)
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/tenacity/__init__.py"", line
349, in iter"
1653590721294, return fut.result()
1653590721294," File
""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 428, in result"
1653590721294, return self.__get_result()
1653590721294," File
""/usr/local/lib/python3.7/concurrent/futures/_base.py"", line 384, in
__get_result"
1653590721294, raise self._exception
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py"",
line 1162, in adopt_or_reset_orphaned_tasks"
1653590721294, to_reset =
self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
1653590721294," File
""/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py"",
line 485, in try_adopt_task_instances"
1653590721294, self.adopted_task_timeouts[ti.key] = ti.queued_dttm +
self.task_adoption_timeout
1653590721294,TypeError: unsupported operand type(s) for +: 'NoneType' and
'datetime.timedelta'
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
Airflow v2.2.3
apache-airflow-providers-http==2.0.0
apache-airflow-providers-microsoft-mssql==2.1.0
apache-airflow-providers-microsoft-winrm==2.0.2
apache-airflow-providers-mongo==2.3.0
apache-airflow-providers-ssh==2.4.0
apache-airflow-providers-celery==2.1.0
### Deployment
Other Docker-based deployment
### Deployment details
Deployed to AWS ECS
### Anything else
We have been using Airflow since version 1.10.2 and never had this issue
before.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
