aehm03 opened a new issue, #23700:
URL: https://github.com/apache/airflow/issues/23700
### Apache Airflow version
2.3.0 (latest released)
### What happened
After updating to 2.3.0 tasks get stuck in the scheduled status. The pool
shows the full number of queued slots, but none of them is running. If I free
up a slot, the following behaviour can be observed in the scheduler/worker logs.
Scheduler:
```
...
[2022-05-13 11:19:26,839] {scheduler_job.py:1218} INFO - Resetting orphaned
tasks for active dag runs
... (Deletion of task) ...
[2022-05-13 11:21:54,866] {base_executor.py:91} INFO - Adding to queue:
['airflow', 'tasks', 'run', 'dag_name', 'task_name',
'scheduled__2022-05-11T05:30:00+00:00', '--local', '--subdir', '...']
[2022-05-13 11:21:54,984] {scheduler_job.py:596} INFO - Executor reports
execution of dag_name.task_name run_id=scheduled__2022-05-11T05:30:00+00:00
exited with status queued for try_number 2
[2022-05-13 11:21:55,043] {scheduler_job.py:630} INFO - Setting external_id
for <TaskInstance: dag_name.task.name scheduled__2022-05-11T05:30:00+00:00
[queued]> to 9ce04a78-b509-4250-8162-e104056ef403
[2022-05-13 11:24:27,678] {scheduler_job.py:1218} INFO - Resetting orphaned
tasks for active dag runs
[2022-05-13 11:29:28,894] {scheduler_job.py:1218} INFO - Resetting orphaned
tasks for active dag runs
[2022-05-13 11:34:29,256] {scheduler_job.py:1218} INFO - Resetting orphaned
tasks for active dag runs
[2022-05-13 11:39:30,138] {scheduler_job.py:1218} INFO - Resetting orphaned
tasks for active dag runs
…
```
Worker:
```
[2022-05-13 11:21:54,877: INFO/MainProcess] Task
airflow.executors.celery_executor.execute_command[9ce04a78-b509-4250-8162-e104056ef403]
received
[2022-05-13 11:21:54,927: WARNING/ForkPoolWorker-15]
/home/airflow/.local/lib/python3.8/site-packages/celery/app/trace.py:660:
RuntimeWarning: Exception raised outside body:
StatementError('(sqlalchemy.exc.ProgrammingError)
(MySQLdb._exceptions.ProgrammingError) (1146, "Table
\'airflow.task_id_sequence\' doesn\'t exist")'):
...
sqlalchemy.exc.StatementError: (sqlalchemy.exc.ProgrammingError)
(MySQLdb._exceptions.ProgrammingError) (1146, "Table 'airflow.task_id_sequence'
doesn't exist")
[SQL: INSERT INTO celery_taskmeta (id, task_id, `status`, result, date_done,
traceback) VALUES (%s, %s, %s, %s, %s, %s)]
[parameters: [{'task_id': '9ce04a78-b509-4250-8162-e104056ef403',
'traceback': None, 'result': None}]]
(Background on this error at: http://sqlalche.me/e/14/f405)
```
So the schedulers are constantly busy resetting orphaned tasks and when a
new task is received, the worker is not able to write into the celery_taskmeta
table. (This could be related or two entirely different problems).
I can not make sense of the error message, I dont't think a table
task_id_sequence should exists and I do not get from the failed SQL statement
why it should exists. My first guess was that the celery_taskmeta table may be
configured wrongly So I adapted the task_id column to match the other task_id
columns, that did not change anything though.
### What you think should happen instead
Tasks should be executed. Workers being able to insert into celery_taskmeta
seems to be related to that.
### How to reproduce
Hard to reproduce, update from 2.1.1. and have a lot of DAGs I guess.
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.1
apache-airflow-providers-docker==2.6.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==6.8.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.8.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.3
### Deployment
Other 3rd-party Helm chart
### Deployment details
* The database migration was quite messy, since the existing columns in
MySQL often had not the correct collation values, I used the offline migration.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
