potiuk commented on issue #15082: URL: https://github.com/apache/airflow/issues/15082#issuecomment-1374651500
The problem (and difficulty) with implementing this one is that you need some central service that would coordinate the rate limits across mutliple parallel running tasks from - potentially - multiple nodes running such client code. Implementing a time-sliding window with request rate by minute (or another time period is not something that could be easily done in "generic" way. There are some projects that implement some generic "services" that can provide such capabilities (Global Distributed Client Side Rate Limiting) - for example https://github.com/youtube/doorman - and you might see there complexities involved in implementing such a solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
