[
https://issues.apache.org/jira/browse/NIFI-14977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042828#comment-18042828
]
Pete Vlad commented on NIFI-14977:
----------------------------------
Hi Pierre,
Thanks again for the update and for implementing {*}NIFI-14978{*}.
I wanted to follow up because the issue is *still occurring in our environment*
even after configuring the StandardWebClientProvider for all Bitbucket registry
clients and forcing *HTTP/1.1* across all of them.
{code:java}
2025-12-04 10:33:28,694 WARN [Replicate Request Thread-47]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/process-groups/d472e1c1-0196-1000-b001-00b31d45f345 to
nifi-cluster-1-node.nifi-cluster-headless.nifi2.svc.cluster.local:8443
org.apache.nifi.web.client.api.WebClientServiceException: Request execution
failed HTTP Method [GET] URI
[https://nifi-cluster-1-node.nifi-cluster-headless.nifi2.svc.cluster.local:8443/nifi-api/flow/process-groups/d472e1c1-0196-1000-b001-00b31d45f345?uiOnly=true]
at
org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.getResponse(StandardWebClientService.java:384)
at
org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.retrieve(StandardWebClientService.java:367)
at
org.apache.nifi.cluster.coordination.http.replication.client.StandardHttpReplicationClient.replicate(StandardHttpReplicationClient.java:204)
at
org.apache.nifi.cluster.coordination.http.replication.client.StandardHttpReplicationClient.replicate(StandardHttpReplicationClient.java:198)
at
org.apache.nifi.cluster.coordination.http.replication.client.StandardHttpReplicationClient.replicate(StandardHttpReplicationClient.java:148)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:641)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:860)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.http.HttpTimeoutException: request timed out
at
java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:953)
at
java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
at
org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.getResponse(StandardWebClientService.java:382)
... 11 common frames omitted
{code}
> NiFi 2.4 Fails to Start Due to Bitbucket Flow Registry Client – “Too
> Many Concurrent Streams” Errors
> -----------------------------------------------------------------------------------------------------
>
> Key: NIFI-14977
> URL: https://issues.apache.org/jira/browse/NIFI-14977
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning, NiFi API
> Affects Versions: 2.4.0
> Environment: Kubernetes, NiFiKop helm chart
> Reporter: Pete Vlad
> Priority: Major
>
> We are running *Apache NiFi 2.4* (NifiKop Helm chart) in a clustered
> environment.
> When NiFi attempts to start up and synchronize with {*}Bitbucket Cloud
> Registry Client{*}, the application fails to fully initialize.
> We have:
> *
> ** *72 process groups* under version control (Bitbucket Cloud registry).
> *
> ** nifi.web.jetty.threads=30 (max web threads).
> *
> ** nifi-atlassian-nar-2.5.0-SNAPSHOT.
> We are observing inconsistent startup behavior:In some cases, NiFi {*}starts
> successfully{*}, but the logs contain repeated errors:
> *java.io.IOException: too many concurrent streams* (from the JDK HTTP/2
> client used by the Bitbucket registry integration).
> Even though these errors appear, the cluster eventually loads, and the UI
> becomes available.
> In other cases, NiFi {*}fails to start entirely{*}. It gets stuck ( The UI is
> visible for a few minutes but then it appear like an empty canvas where you
> cannot change or create anything) at the following log entry and does not
> progress further:
> {code:java}
> 2025-09-04 13:57:41,246 INFO [Framework Task Thread-27]
> o.a.n.a.b.BitbucketFlowRegistryClient BitbucketFlowRegistryClient[id=XXXXXX]
> Initializing repository client{code}
> When this occurs, the Controller Settings → Registry Clients page cannot
> load, and NiFi remains blocked.
> If we disable the Bitbucket registry (by modifying the registry client URL so
> it cannot connect), NiFi starts consistently without issue.
> We need guidance on:
> # Whether this is a {*}known issue in NiFi 2.4’s Bitbucket Flow Registry
> Client{*}.
> # Whether there is a *configuration option* to force HTTP/1.1 or limit
> HTTP/2 concurrency for registry communications. (Not sure if this is the case)
> # Whether there is a *supported workaround* to prevent registry client
> initialization from blocking NiFi startup when many versioned process groups
> are present.
> h3. *Logs when NiFi Starts Successfully (but with Errors)*
> Example log with *“too many concurrent streams”* during startup:
> {code:java}
> at
> org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.getResponse(StandardWebClientService.java:366)
> at
> org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.retrieve(StandardWebClientService.java:349)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:251)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:237)
> at jdk.proxy26/jdk.proxy26.$Proxy234.retrieve(Unknown Source)
> at
> org.apache.nifi.atlassian.bitbucket.BitbucketRepositoryClient.getPagedResponseValues(BitbucketRepositoryClient.java:373)
> at
> org.apache.nifi.atlassian.bitbucket.BitbucketRepositoryClient.getListCommits(BitbucketRepositoryClient.java:362)
> at
> org.apache.nifi.atlassian.bitbucket.BitbucketRepositoryClient.getLatestCommit(BitbucketRepositoryClient.java:403)
> at
> org.apache.nifi.atlassian.bitbucket.BitbucketRepositoryClient.getContentFromBranch(BitbucketRepositoryClient.java:244)
> at
> org.apache.nifi.registry.flow.git.AbstractGitFlowRegistryClient.getSnapshot(AbstractGitFlowRegistryClient.java:491)
> at
> org.apache.nifi.registry.flow.git.AbstractGitFlowRegistryClient.getFlow(AbstractGitFlowRegistryClient.java:279)
> at
> org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.lambda$getFlow$6(StandardFlowRegistryClientNode.java:231)
> at
> org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.execute(StandardFlowRegistryClientNode.java:314)
> at
> org.apache.nifi.registry.flow.StandardFlowRegistryClientNode.getFlow(StandardFlowRegistryClientNode.java:231)
> at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeWithFlowRegistry(StandardProcessGroup.java:3769)
> at
> org.apache.nifi.groups.StandardProcessGroup.lambda$setVersionControlInformation$27(StandardProcessGroup.java:3575)
> at
> org.apache.nifi.controller.scheduling.StandardProcessScheduler.lambda$wrapTask$1(StandardProcessScheduler.java:177)
> at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
> Caused by: java.io.IOException: too many concurrent streams
> at
> java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:967)
> at
> java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
> at
> org.apache.nifi.web.client.StandardWebClientService$StandardHttpRequestBodySpec.getResponse(StandardWebClientService.java:364)
> ... 19 common frames omitted
> Caused by: java.io.IOException: too many concurrent streams
> at
> java.net.http/jdk.internal.net.http.Http2Connection.reserveStream0(Http2Connection.java:616)
> at
> java.net.http/jdk.internal.net.http.Http2Connection.reserveStream(Http2Connection.java:583)
> at
> java.net.http/jdk.internal.net.http.Http2ClientImpl.getConnectionFor(Http2ClientImpl.java:109)
> at java.net.http/jdk.internal.net.http.ExchangeImpl.get(ExchangeImpl.java:94)
> at
> java.net.http/jdk.internal.net.http.Exchange.establishExchange(Exchange.java:391)
> at
> java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl0(Exchange.java:584)
> at
> java.net.http/jdk.internal.net.http.Exchange.responseAsyncImpl(Exchange.java:428)
> at
> java.net.http/jdk.internal.net.http.Exchange.responseAsync(Exchange.java:420)
> at
> java.net.http/jdk.internal.net.http.MultiExchange.responseAsyncImpl(MultiExchange.java:414)
> at
> java.net.http/jdk.internal.net.http.MultiExchange.lambda$responseAsync0$2(MultiExchange.java:347)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
> at
> java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
> at
> java.base/java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2719)
> at
> java.net.http/jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:300)
> at
> java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:1050)
> at
> java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:931)
> ... 21 common frames omitted {code}
> What worked was creating a StandardWebClientProvider for each Bitbucket
> registry. However, I’m now running into the same issue again: when too many
> calls are made too quickly to the NiFi API, I hit the *“too many concurrent
> streams”* error. This causes the UI to become unusable and displays the
> following error:
> {code:java}
> Failed to load Flow Configuration. -
> [org.apache.nifi.web.client.api.WebClientServiceException: Request execution
> failed HTTP Method [GET] URI
> [https://nifi-cluster-3-node.nifi-cluster-headless.nifi2.svc.cluster.local:8443/nifi-api/flow/config]]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
