keon94 commented on issue #3123: URL: https://github.com/apache/incubator-devlake/issues/3123#issuecomment-1296406058
The Github tap does provide rate-limiting support upon closer look. The logic is [here](https://github.com/singer-io/tap-github/blob/8973411ee2f2cf007fe74adcae203f116750180f/tap_github/client.py) called from [here|(https://github.com/singer-io/tap-github/blob/8973411ee2f2cf007fe74adcae203f116750180f/tap_github/client.py#L201)]. There's an undocumented config called "max_sleep_seconds" which defines the upper bound of "X-RateLimit-Reset" (wait-time until next API call). We can add that to the config JSON to override its default of 600s. Simply put, if we exceed the rate limit, the tap sleeps for up to this many seconds before issuing the next API call. It'll fail if it has to sleep for longer. cc @hezyin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
