rajrao opened a new issue, #36539:
URL: https://github.com/apache/airflow/issues/36539

   ### Description
   
   Currently when one pauses a DAG, it allows the currently executing task to 
complete and then leaves all downstream tasks to go to scheduled.
   I feel like there should be the ability to "Complete DAG run and then 
Pause", so that the entire DAG run can complete.
   
   ### Use case/motivation
   
   Often times we need to take down a certain service that a DAG is dependent 
on (eg: database server). Currently, we have the ability to pause the dag, 
which allows the executing task to complete and then puts all other downstream 
tasks in "scheduled" state and runs them once the Dag is unpaused.
   
   When the DAG is long running, we ideally would like to allow the entire DAG 
to complete so that the result of the DAG can be made available for customers 
and then pause the DAG. This way, I could put all these dags into "Complete and 
then Pause" mode and wait for all the work against my services to complete. 
After that I could take them offline, or do whatever work on them and then 
unpause all of the DAGs.
   
   Another use case is when we have to upgrade the Airflow. In such cases, 
again it would be nice to get all DAGs that are running to complete and then 
upgrade the Airflow.
   
   In the absence of such functionality, we are looking at adding a 
shortcircuit operator to our DAGs that would then check if a variable is set 
and if it is, then it would return False and short-circuit then run. This seems 
like adding an additional bit of plumbing that might be best implemented 
directly as part of DAG UI via an option like "Complete all downstream tasks 
and then Pause"
   
   For the Airflow upgrade use case, often times we have to take longer for the 
upgrade, as someone has to look at the DAG runs and as the complete, pause them 
and once they are all paused, then do the upgrade. And this needs someone to 
babysite Airflow. But, if there was a single option to allow ALL dags to 
"complete and then pause" this would be so much better.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to