rubenbriones commented on issue #15082:
URL: https://github.com/apache/airflow/issues/15082#issuecomment-1374481315

   That would be really nice. I have multiple ETLs that scrape data from Free 
API that have a max rate limit requests **by minute**, and I haven't figure a 
way to implement this rate limit logic in Airflow.
   
   I'm using Dynamic Task Mapping for generating 1000 tasks (to scrape 1000 
different items data, from the same API), but i want to schedule them in a way 
that the rate limit is not surpassed. 
   
   Maybe do you think it is possible to implement this with a custom Deferred 
HttpOperator, that before sending the HTTP requests checks (via XCom) how many 
requests have been sent over the last X minutes to the http_conn_id requested? 
That new custom operator should update the XCom cache after making new 
requests. ¿That makes sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to