Julien Ledoux created NIFI-14369:
------------------------------------
Summary: GitHubFlowRegistryClient reaches GitHub API quotas limits
Key: NIFI-14369
URL: https://issues.apache.org/jira/browse/NIFI-14369
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
Environment: Official Docker Images 2.0.0 stable running on Ubuntu
22.04.4 LTS and Docker v27.0.3
Reporter: Julien Ledoux
When using GitHubFlowRegistryClient with a GitHub App installation token and
more than 10 versioned workflows, you may frequently encounter an issue where
GitHub stops responding for no apparent reason. In such cases, the NiFi
Registry dialog simply displays a red error message like "[object Object]".
To diagnose the issue, you'll need to check the nifi-user.log file. However,
since its content is not sent to the Docker container's stdout, you'll need to
mount the logs folder in a volume to access it locally, or login into the
docker container.
Upon inspecting the logs, you’ll likely find the following error message:
{code:java}
2025-02-27 10:27:57,187 WARN [NiFi Web Server-271]
o.a.n.w.a.config.NiFiCoreExceptionMapper org.apache.nifi.web.NiFiCoreException:
Unable to get branches for registry with ID
bcb1f2db-0194-1000-e6ec-53e8c29cfce8: {"message":"API rate limit exceeded for
installation ID xxxxxxxx. If you reach out to GitHub Support for help, please
include the request ID 27EC:10FD3E:2CFBDC2:2EAF364:67C02FDE and timestamp
2025-02-27 09:26:57
UTC.","documentation_url":"https://docs.github.com/rest/overview/rate-limits-for-the-rest-api","status":"403"}.
Returning Conflict response.
org.apache.nifi.web.NiFiCoreException: Unable to get branches for registry with
ID bcb1f2db-0194-1000-e6ec-53e8c29cfce8: {"message":"API rate limit exceeded
for installation ID xxxxxxxx. If you reach out to GitHub Support for help,
please include the request ID 27EC:10FD3E:2CFBDC2:2EAF364:67C02FDE and
timestamp 2025-02-27 09:26:57
UTC.","documentation_url":"https://docs.github.com/rest/overview/rate-limits-for-the-rest-api","status":"403"}
at
org.apache.nifi.web.dao.impl.StandardFlowRegistryDAO.getBranchesForUser(StandardFlowRegistryDAO.java:133)
at
org.apache.nifi.web.StandardNiFiServiceFacade.getBranches(StandardNiFiServiceFacade.java:3285)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
at
org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:161)
at
org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:120)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:637)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:627)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
at
org.apache.nifi.web.StandardNiFiServiceFacade$$SpringCGLIB$$0.getBranches(<generated>)
at org.apache.nifi.web.api.FlowResource.getBranches(FlowResource.java:2344)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:397)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:349)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at
org.eclipse.jetty.ee10.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1379)
at
org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)
at
org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
at
org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.apache.nifi.web.security.log.AuthenticationUserFilter.doFilterInternal(AuthenticationUserFilter.java:57)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:94)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:56)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:128)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:94)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:56)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.apache.nifi.web.security.csrf.CsrfCookieFilter.doFilterInternal(CsrfCookieFilter.java:43)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.apache.nifi.web.security.csrf.SkipReplicatedCsrfFilter.doFilterInternal(SkipReplicatedCsrfFilter.java:59)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278)
at
org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at
org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:44)
at
org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:208)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at
org.eclipse.jetty.ee10.servlets.DoSFilter.doFilterChain(DoSFilter.java:462)
at
org.apache.nifi.web.server.filter.DataTransferExcludedDoSFilter.doFilterChain(DataTransferExcludedDoSFilter.java:51)
at org.eclipse.jetty.ee10.servlets.DoSFilter.doFilter(DoSFilter.java:317)
at org.eclipse.jetty.ee10.servlets.DoSFilter.doFilter(DoSFilter.java:282)
at
org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at
org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at
org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
at
org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
at
org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:824)
at
org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:436)
at
org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
at
org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:717)
at
org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:181)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:151)
at org.eclipse.jetty.server.Server.handle(Server.java:182)
at
org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:662)
at
org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:414)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
at
org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
at
org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.kohsuke.github.HttpException: {"message":"API rate limit
exceeded for installation ID xxxxxxxx. If you reach out to GitHub Support for
help, please include the request ID 27EC:10FD3E:2CFBDC2:2EAF364:67C02FDE and
timestamp 2025-02-27 09:26:57
UTC.","documentation_url":"https://docs.github.com/rest/overview/rate-limits-for-the-rest-api","status":"403"}
at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:740)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:480)
at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:146)
at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:93)
at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:116)
at org.kohsuke.github.PagedIterator.nextPageArray(PagedIterator.java:144)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:85)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:113)
at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:2626)
at
org.apache.nifi.github.GitHubRepositoryClient.lambda$getBranches$1(GitHubRepositoryClient.java:195)
at
org.apache.nifi.github.GitHubRepositoryClient.execute(GitHubRepositoryClient.java:421)
at
org.apache.nifi.github.GitHubRepositoryClient.getBranches(GitHubRepositoryClient.java:195)
at
org.apache.nifi.registry.flow.git.AbstractGitFlowRegistryClient.getBranches(AbstractGitFlowRegistryClient.java:156)
at
org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.lambda$getBranches$0(StandardFlowRegistryClientNode.java:201)
at
org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.execute(StandardFlowRegistryClientNode.java:314)
at
org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.getBranches(StandardFlowRegistryClientNode.java:201)
at
org.apache.nifi.web.dao.impl.StandardFlowRegistryDAO.getBranchesForUser(StandardFlowRegistryDAO.java:120)
... 127 common frames omitted
Caused by: java.io.InterruptedIOException: null
at org.kohsuke.github.RateLimitHandler$1.onError(RateLimitHandler.java:83)
at org.kohsuke.github.RateLimitHandler.onError(RateLimitHandler.java:50)
at org.kohsuke.github.GitHubClient.detectKnownErrors(GitHubClient.java:497)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:464)
... 142 common frames omitted
Caused by: org.kohsuke.github.HttpException: API rate limit reached
at org.kohsuke.github.RateLimitHandler.onError(RateLimitHandler.java:49)
... 144 common frames omitted {code}
To prevent this issue—and avoid waiting an hour before being able to push or
pull from GitHub—it would be beneficial to have a way to adjust the GitHub API
request rate when handling a large number of workflows. Alternatively, a manual
synchronization button to trigger workflow updates from GitHub could also be a
useful way to fix this behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
