[
https://issues.apache.org/jira/browse/NIFI-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654885#comment-15654885
]
ASF GitHub Bot commented on NIFI-2996:
--------------------------------------
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/1192
Thanks for the PR @mosermw! I was just running it and had some follow-up
questions. It looks as though the proposed changes are conditionally validating
the underlying Processor when it is stopped. However, other validation about
the Processor (specifically regarding the incoming and outgoing connections) is
still happening. Comments about these changes are outlined below according to
the modified method.
**isValid**
isValid is used throughout the application to check processor state usually
prior to performing some action or transitioning to some state. I believe in
this method we still need to run all Processor validation logic to ensure we
know if it's valid when invoking it.
When looking into the context of the isValid usage, it does appear that we
can make some changes when getting the Processor status [1]. By checking if the
scheduled state is running prior to checking the processor validity aligns with
the proposed solution and should help reduce the expense of the status calls
identified in the JIRA.
**getValidationErrors**
Regardless of whether the UI is rendering the validation results, they are
still being returned from the Rest Api. I'm wondering if returning a portion of
the validation errors when the Processor is not stopped is confusing. Would it
be more clear if we moved the rest of the validation inside of the conditional
as well? Meaning, when the Processor is not stopped, this method would return
an empty Collection. When the Processor is stopped, this method would run all
Processor validation.
**Additional Comment**
Should these changes also be applied to Ports, Reporting Tasks, and
Controller Services (if disabled)? I understand that not all of these
components cause lengthy validation times, but I'm trying to consider it from a
consistency perspective from a client of the Rest Api. If a Processor only
returns validation errors when it is stopped, I believe this is how all
components should work too.
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L2704
> Processor/Service validation takes exponentially longer to run as the graph
> grows
> ---------------------------------------------------------------------------------
>
> Key: NIFI-2996
> URL: https://issues.apache.org/jira/browse/NIFI-2996
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0, 0.7.1
> Reporter: Michael Moser
> Assignee: Michael Moser
> Fix For: 1.1.0
>
>
> As you add processors that reference controller services to your NiFi, the
> validation that occurs during normal UI usage increases dramatically.
> When running in a cluster, I have to increase the nifi.properties
> nifi.cluster.node.read.timeout well beyond its 5 sec default timeout in order
> for the UI to work. Eventually, simple operations in the UI take close to a
> minute to happen.
> As a test, I created an SSLContextService using certs created with the
> amazingly useful nifi-toolkit. I created a DistributedMapCacheClientService
> that references this SSLContextService. Then I created 108
> FetchDistributedMapCache processors that reference the
> DistributedMapCacheClient service. I used ${hostname(true)} for my
> FetchDistributedMapCache processor's Cache Entry Identifier property.
> When NiFi is up with no UI connected, during a single "NiFi status" phase I
> noticed that the SSLContextService was validated 108 times and the EL
> hostname(true) was evaluated 216 times. When I connect the UI and go through
> a normal status refresh cycle, the SSLContextService was validated 432 times
> and the EL hostname(true) was evaluated 864 times. These validations take a
> full second on my slowest machine.
> In NiFi 0.x, the expensive REST API call is
> /nifi-api/controller/controller-services/node.
> In NiFi 1.x, it appears to divide the work among REST API calls to
> /nifi-api/process-groups/UUID and
> /nifi-api/flow/process-groups/UUID/controller-services and
> /nifi-api/flow/status
> Rather than attempting to fix StandardSSLContextService or the EL
> HostnameEvaluator, I wonder if there was a more generic approach to helping
> resolve this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
