gorros commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-574622784 @aviemzur This is a great feature. Thanks for contribution. Currently I have couple of long term running clusters and i used to do same what you do. I was listing clusters and filtering by name. But that was resulted in throttling exception from AWS API when number of such dags was growing. Then I started to persist id of currently active cluster. Before submitting a step I was describing the cluster and making sure it is still running. This reduced number of throttling exceptions but still there were some. Then I introduced exception handling and [exponential back off ](https://stackoverflow.com/questions/27438273/exponential-backoff-time-sleep-with-random-randint0-1000-1000). This final step eliminated throttling issue. Also does your implementation take care of listing cluster with pagination?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
