EinavDanielDX commented on issue #35610:
URL: https://github.com/apache/airflow/issues/35610#issuecomment-1879997134
I have update on the matter.
More background regarding the deployment:
We are using airflow with Celery Executor on Kubernetes, in addition, as we
don't need the workers up and running all the time we are using Keda for
scaling up and down the workers, which means workers are being killed regularly.
Our deployment is configured to allow graceful shutdown for 4 hours ( which
is more than enough for any task we run to finish its processing).
We noticed that beside the misalignment between successful jobs and failed
celery tasks we have other errors on random tasks, such as
"Executor reports task instance finished (success) although the task says
it's queued. (Info: None) Was the task killed externally?"
or
"Loading metadata failed: {'success': False, 'reason':
AirflowException('Task received SIGTERM signal')}"
After further investigation set
`AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False"`
Now our flower looks better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
