I am writing a client/crawler/scraper library for a certain website, hopefully
my first nimble package. The site has some rate limiting which I haven't figure
out exactly yet, but will try to do. Once I do, I want to have a request
scheduler so that all requests (async) done to the website are automatically
delayed by the right amount of time once I reach the rate limit threshold value
(eg, 1000 requests per hour, 30 requests per minute etc), so that it doesn't
trip the rate limiting. I am thinking a template to wrap the procedures call
bodies in and send it to the schedueler would be a proper interface. Or maybe I
should just compute the required sleep time value (if needed) at each call, and
make a proc that wraps the call to the proc, adding a sleepAsync call before it
? I feel this is a rather common use case, and we should have a library for
this. I could only find this after 5 minutes of searching:
<https://nimble.directory/pkg/taskqueue> \- but I can't say I understand how it
works/how to use it after reading the readme.
