maxcountryman commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-924064575
We're continuing to see an issue in `2.1.3` where DAG tasks appear to be running for many hours, even days, but are making no visible progress. This situation persists through scheduler restarts and is not resolved until we clear each stuck task manually twice. The first time we clear the task we see an odd rendering in the UI with a dark blue border that looks like this: <img width="637" alt="Screen Shot 2021-09-21 at 7 40 43 AM" src="https://user-images.githubusercontent.com/74351/134193377-7f826066-37cc-4f48-bb6f-b2b2224a6be7.png";> The second time we clear the task the border changes to light green and it usually completes as expected. This is a pretty frustrating situation because the only known path to remediation is manual intervention. As previously stated we're deploying to ECS, with each Airflow process as its own ECS service in an ECS cluster; this generally works well except as noted here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/74351/134193377-7f826066-37cc-4f48-bb6f-b2b2224a6be7.png"](https://user-images.githubusercontent.com/74351/134193377-7f826066-37cc-4f48-bb6f-b2b2224a6be7.png"){kind=link}