rubenbriones commented on issue #15082: URL: https://github.com/apache/airflow/issues/15082#issuecomment-1374481315
That would be really nice. I have multiple ETLs that scrape data from Free API that have a max rate limit requests **by minute**, and I haven't figure a way to implement this rate limit logic in Airflow. I'm using Dynamic Task Mapping for generating 1000 tasks (to scrape 1000 different items data, from the same API), but i want to schedule them in a way that the rate limit is not surpassed. Maybe do you think it is possible to implement this with a custom Deferred HttpOperator, that before sending the HTTP requests checks (via XCom) how many requests have been sent over the last X minutes to the http_conn_id requested? That new custom operator should update the XCom cache after making new requests. ¿That makes sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
