potiuk commented on issue #13495: URL: https://github.com/apache/airflow/issues/13495#issuecomment-754824918
This cannot be done on operator level due to the way operators work in Airflow. However there is a feature in Airflow that is designed to handle this case - pools. You should experiment what is the maximum number of running operators that will not cause exceeding quotas and set the maximum size of the pool to that value. If you then use the pool for all your DAGs wehen This way you will have maxiumum N runing tasks for the service at a time - no matter how many of those are queued. You can read more about pools here: http://airflow.apache.org/docs/apache-airflow/1.10.14/concepts.html#pools ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
