[JIRA] (JENKINS-41112) GitHub Branch Source should throttle calls to stay below rate limit

[bat...@batmat.net (JIRA)]

Title: Message Title

Baptiste Mathus created an issue   Jenkins / JENKINS-41112 GitHub Branch Source should throttle calls to stay below rate limit Issue Type: Improvement Assignee: Unassigned Components: github-branch-source-plugin Created: 2017/Jan/16 9:01 PM Priority: Critical Reporter: Baptiste Mathus As a user, I would like to be able to use GitHub Branch Source plugin on big GitHub organizations. Right now, though it's a lot better, we still hit the wall of the rate limit. The workaround could be to specify a Repository name pattern but that wouldn't work for all our cases because we don't have naming rules that would enable that. I think the plugin could for example: check the number of repositories of the requested organization (and the given pattern) do a rough calculation of the number of API calls warn the user it's going to take (far) more time than pure CPU/network capacities should enable to be able to complete the operation. Example: say I need 10 API calls per repo and I find the user has 1000. This would be 10k calls, instead of the max of 5k. Without even counting the probable other calls coming from some other plugins and so on. So, the plugin would regularly sleep from time to time to throttle and stay under the radar. Here is an example of what the scanning provoked in term of API calls after I created a new Org Folder and clicked on Save: $ while true do curl -s -u USER:REDACTED_PAT https://api.github.com/rate_limit | jq -r .rate.remaining sleep 10 done 4996 4996 4742 4518 4316 4072 3884 3707 3320 2800 2211 1895 1685 1466 1231 952 685 454 204 0 0 And sure, when 0 was reached, the plugin started failing as expected: Started [Mon Jan 16 20:50:34 UTC 2017] Starting organization scan... [Mon Jan 16 20:50:34 UTC 2017] Updating actions... Looking up details of ORG... ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not refresh actions for navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181 org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259) at org.kohsuke.github.Requester.handleApiError(Requester.java:649) at org.kohsuke.github.Requester._to(Requester.java:284) at org.kohsuke.github.Requester.to(Requester.java:225) at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.retrieveActions(GitHubSCMNavigator.java:476) at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:315) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:371) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) [Mon Jan 16 20:50:34 UTC 2017] Consulting GitHub Organization ERROR: [Mon Jan 16 20:50:34 UTC 2017] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@1164b181 org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259) at org.kohsuke.github.Requester.handleApiError(Requester.java:649) at org.kohsuke.github.Requester._to(Requester.java:284) at org.kohsuke.github.Requester.to(Requester.java:225) at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) [Mon Jan 16 20:50:34 UTC 2017] Finished organization scan. Scan took 0.24 sec FATAL: Failed to recompute children of Plugins » Operations center all the things org.jenkinsci.plugins.github_branch_source.RateLimitExceededException: GitHub API rate limit exceeded at org.jenkinsci.plugins.github_branch_source.Connector$1.onError(Connector.java:259) at org.kohsuke.github.Requester.handleApiError(Requester.java:649) at org.kohsuke.github.Requester._to(Requester.java:284) at org.kohsuke.github.Requester.to(Requester.java:225) at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:669) at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:265) at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:420) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:187) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:139) at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:814) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Finished: FAILURE Add Comment   This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.