GitHub user potiuk added a comment to the discussion: Airflow ci/cd pipeline to 
rebuild airflow

I think you need to trace down what happens and what error you have - there is 
no "one way" - especially that docker-compose is not something we have 
recommended or have a guildeline to on how to productionize it, so any scripts 
you do, you need to make sure you test. I guess you mist have some error - say 
for example your docker compose down will fail (not sure how docker compose 
works) if there are no containers and that might break your whole command. 

Not sure also what "endless shutdown means" and why - this is only you can see 
on what stops your compose from shutting down.  That needs you to login to the 
server and see what's going on when you do "docker compose down" - there might 
be multiple reasons.

Re - pausing and unpausing dags - you could - potentially utilise CLI (as 
@enchant3dmango mentioned) or - better - REST API and python client we release, 
to iterate through and pause or maybe stop all the running tasks.

Again note that docker compose is not a perfect way of deploying airlfow as it 
lacks a lot of flexibility of things like K8S executor for example. Where you 
can shutdown and restart airflow without actually stopping all running pods 
(Airlfow has "pod adoption" mechanism in this case. So it's a bit of your chose 
to choose a platform (docker-compose) that has very little support for more 
"prodution" support you would like to see, and requiring from you to do more 
custom work to make it happen.

GitHub link: 
https://github.com/apache/airflow/discussions/47149#discussioncomment-12340604

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to