ROOBALJINDAL commented on issue #67272:
URL: https://github.com/apache/airflow/issues/67272#issuecomment-4508510559
No, we are not using deferrable property.
fyi..
**Update:**
We logged a ticket with AWS guys too and they responded
```
"I would like to inform you that this is a known issue with Celery in open
source Airflow in versions 3.0.X that causes tasks to fail mid-execution
without logs.The issue is caused by Celery using fork to create new processes
to run tasks. This operation is not safe and can lead to SIGKILL and thread
deadlock."
Workaround:
Update MWAA config as per the node type
===========
Small celery.worker_autoscale 5,5
Medium celery.worker_autoscale. 10,10
Large. celery.worker_autoscale. 20,20
```
We tried this but still faced the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
