taylorfinnell commented on pull request #17819:
URL: https://github.com/apache/airflow/pull/17819#issuecomment-925228694
@ephraimbuddy @kaxil
We applied this patch directly on top of the 2.1.4 tag and noticed issues
almost instantly.
- Tasks would queue
- Tasks would throw an exception - the exception in this case was from our
DAG code signaling we are not ready to process, so retry later. It was not an
internal airflow exception
- Task logs would indicate that state was being set to `UP_FOR_RETRY`
- The UI would show the task still `RUNNING`. The task would have ~16
retries left to go and would sit on attempt 1
- Clearing the task would result in `shutdown` state
- To get the task to run again we first had to fail it, then clear it again
We also tried the `main` branch up to this commit and saw very similar
issues.
Here are some logs.
```
...
[2021-09-22 04:47:15,214] {taskinstance.py:1463} ERROR - Task failed with
exception
...
raise FileNotFoundError(f"{file} does not exist")
[2021-09-22 04:47:15,299] {logging_mixin.py:109} WARNING -
/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/strategies.py:911
SAWarning: Multiple rows returned with uselist=False for lazily-loaded
attribute 'DagRun.task_instances'
[2021-09-22 04:47:15,301] {taskinstance.py:1512} INFO - Marking task as
UP_FOR_RETRY. dag_id=foo_dag_split, task_id=foo_task,
execution_date=20210921T043000, start_date=20210922T044714,
end_date=20210922T044715
[2021-09-22 04:47:15,302] {logging_mixin.py:109} WARNING -
/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py:2193
SAWarning: Instance <TaskInstance at 0x7f01f5adc070> is already pending in this
Session yet is being merged again; this is probably not what you want to do
[2021-09-22 04:47:15,367] {local_task_job.py:151} INFO - Task exited with
return code 1
[2021-09-22 04:47:15,517] {logging_mixin.py:109} WARNING -
/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/strategies.py:911
SAWarning: Multiple rows returned with uselist=False for lazily-loaded
attribute 'DagRun.task_instances'
[2021-09-22 04:47:15,517] {taskinstance.py:1512} INFO - Marking task as
UP_FOR_RETRY. dag_id=foo_dag_split, task_id=foo_task,
execution_date=20210921T043000, start_date=20210922T044714,
end_date=20210922T044715
[2021-09-22 04:47:15,518] {logging_mixin.py:109} WARNING -
/opt/app-root/lib64/python3.8/site-packages/sqlalchemy/orm/session.py:2193
SAWarning: Instance <TaskInstance at 0x7f01f5ad6580> is already pending in this
Session yet is being merged again; this is probably not what you want to do
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
