[
https://issues.apache.org/jira/browse/NIFI-11814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-11814:
------------------------------
Attachment: NIFI-11814-thread-dumps.txt
> Controller Services that are enabling (not enabled) appear to block parameter
> context changes
> ---------------------------------------------------------------------------------------------
>
> Key: NIFI-11814
> URL: https://issues.apache.org/jira/browse/NIFI-11814
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.21.0, 1.22.0
> Reporter: Joe Witt
> Priority: Blocker
> Fix For: 1.23.0
>
> Attachments: NIFI-11814-thread-dumps.txt
>
>
> Flagging as a blocker for given the problem it can cause.
> Scenario:
> * User has a significant number of Controller Services (more than 1000 of
> them)
> * Some Controller Services reference Param Context values some dont.
> * A handful of the Controller Services are failing to start but are not
> disabled. They're invalid but listed as 'enabling' when looking at
> diagnostics for instance. Such as
> {noformat}
> DBCPConnectionPool : 1003 total, {enabling=1, enabled=278, disabled=724}
> {noformat}
> * Now user tries to edit a parameter in a parameter context that does not
> reference any controller services (or they make a new param context then try
> to add a param at all). It will timeout after about 30 seconds failing due
> to
> {noformat}
> 2023-07-14 14:21:13,791 WARN [Replicate Request Thread-219480]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/process-groups/root/controller-services to host:9443 due to
> java.net.ProtocolException: unexpected end of stream
> 2023-07-14 14:21:13,791 WARN [Replicate Request Thread-219480]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> java.net.ProtocolException: unexpected end of stream
> at
> okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.kt:415)
> at
> okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276)
> at okio.RealBufferedSource.read(RealBufferedSource.kt:189)
> at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:197)
> at okio.InflaterSource.refill(InflaterSource.kt:112)
> at okio.InflaterSource.readOrInflate(InflaterSource.kt:76)
> at okio.InflaterSource.read(InflaterSource.kt:49)
> at okio.GzipSource.read(GzipSource.kt:69)
> at okio.Buffer.writeAll(Buffer.kt:1290)
> at okio.RealBufferedSource.readByteArray(RealBufferedSource.kt:236)
> at okhttp3.ResponseBody.bytes(ResponseBody.kt:124)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.getResponseBytes(OkHttpReplicationClient.java:168)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:138)
> at
> org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:130)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:645)
> at
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:869)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2023-07-14 14:21:13,791 ERROR [Parameter Context Update Thread-2]
> o.a.n.web.api.ParameterContextResource Failed to update Parameter Context
> org.apache.nifi.web.util.LifecycleManagementException: Failed while waiting
> for Controller Services to finish transitioning to a state of DISABLED
> at
> org.apache.nifi.web.util.ClusterReplicationComponentLifecycle.activateControllerServices(ClusterReplicationComponentLifecycle.java:469)
> at
> org.apache.nifi.web.util.ParameterUpdateManager.disableControllerServices(ParameterUpdateManager.java:273)
> at
> org.apache.nifi.web.util.ParameterUpdateManager.updateParameterContexts(ParameterUpdateManager.java:144)
> at
> org.apache.nifi.web.api.ParameterContextResource.lambda$submitUpdateRequest$17(ParameterContextResource.java:832)
> at
> org.apache.nifi.web.api.concurrent.AsyncRequestManager$2.run(AsyncRequestManager.java:117)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java
> {noformat}
> * Ironically then if you look at why some of the controller services failed
> to enable it can be for reasons like a parameter they reference not being
> available. Which you could not go fix as that would then timeout due to the
> enabling controller service. For example
> {noformat}
> 2023-07-14 22:21:26,912 ERROR [Timer-Driven Process Thread-4]
> o.a.n.c.s.StandardControllerServiceNode Unable to enable
> StandardControllerServiceNode[service=DBCPConnectionPool[id=someuuid],
> name=ENEL-DBCPConnectionPool, active=true]. Last known validation state was
> INVALID : ['Database Connection URL' is invalid because Property references
> Parameter 'hostname' but the currently selected Parameter Context does not
> have a Parameter with that name, 'Database Connection URL' is invalid because
> Property references Parameter 'DB_Name' but the currently selected Parameter
> Context does not have a Parameter with that name, 'Database User' is invalid
> because Property references Parameter 'DB_User' but the currently selected
> Parameter Context does not have a Parameter with that name, 'Password' is
> invalid because Property references Parameter 'DB_Password' but the currently
> selected Parameter Context does not have a Parameter with that name]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
