Hey guys, I have been successfully using airflow on AWS ECS for a while now. I have about 30 DAGs that run on a daily basis. Recently, I noticed that one of my DAGs has been getting stuck in running state. When I checked the DAG tasks, I noticed that that one of the downstream tasks just didn't queue up even though it's upstream task was completed successfully. When I restarted the ECS container, the tasks continued to queue up and run normally.
I think there is a resource problem where too much memory/cpu is being used prohibiting the scheduler from performing its duty normally. In summary, how do I make it so that airflow uses less resources, or that it frees up resources so the scheduler can run properly? I am using LocalExecutor by the way on AIrflow 1.8.2. Thanks! - Ali
