[
https://issues.apache.org/jira/browse/NIFI-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Secules updated NIFI-14712:
--------------------------------
Description:
Background: I have a very large flow with 50000 or so processors and the
flow-analysis endpoint is taking 20 or more seconds to complete. This is likely
because the endpoint is traversing all the process groups recursively each time
it is called. While User A had the flow canvas open to the root node, User B
was unable to make configuration changes to the flow and also had a very hard
time making updates to the flow.
This endpoint also holds a read lock on the flow for the entire time, so it
will prevent any writes.
Suggestions:
# Short circuit this endpoint and return a default success value if there are
no violations at all.
# Only show detailed validation errors for the current process group and not
descendants in the UI. For process groups within the current process group,
show a cumulative count of violations within each process group. Counts would
be updated lazily.
# The UI only checks whether there exist violations at the current process
group and descendants, only load the detailed validation errors if the user
clicks on the first aid kit icon. A map of process groupID to boolean would be
updated at the same time violations are updated and a violation on a process
group would set the boolean to true for its ancestors up to the root.
## This can be tricky because process groups don't know their parent already.
Also I think changing the meaning of the "first aid kit" to only correspond to
the root PG, while it would help this performance situation, it would be less
intuitive for the user if they're on any other process group than the root.
# Keep a read through cache of PG id to violations. Empty it whenever the
primary map of rule violations changes. On the validatior side, add exponential
sleep between consecutive validation passes until the flow stops being modified.
Screenshots from browser dev tools attached
was:
Background: I have a very large flow with 50000 or so processors and the
flow-analysis endpoint is taking 20 or more seconds to complete. This is likely
because the endpoint is traversing all the process groups recursively each time
it is called. While User A had the flow canvas open to the root node, User B
was unable to make configuration changes to the flow and also had a very hard
time making updates to the flow.
This endpoint also holds a read lock on the flow for the entire time, so it
will prevent any writes.
Suggestions:
# Short circuit this endpoint and return a default success value if there are
no violations at all.
# Only show detailed validation errors for the current process group and not
descendants in the UI. For process groups within the current process group,
show a cumulative count of violations within each process group. Counts would
be updated lazily.
# The UI only checks whether there exist violations at the current process
group and descendants, only load the detailed validation errors if the user
clicks on the first aid kit icon. A map of process groupID to boolean would be
updated at the same time violations are updated and a violation on a process
group would set the boolean to true for its ancestors up to the root.
## This can be tricky because process groups don't know their parent already.
Also I think changing the meaning of the "first aid kit" to only correspond to
the root PG, while it would help this performance situation, it would be less
intuitive for the user if they're on any other process group than the root.
Screenshots from browser dev tools attached
> Performance issue with flow-analysis for a process group
> --------------------------------------------------------
>
> Key: NIFI-14712
> URL: https://issues.apache.org/jira/browse/NIFI-14712
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.4.0
> Reporter: Eric Secules
> Priority: Major
> Attachments: Screenshot 2025-07-03 at 7.04.37 PM.png, Screenshot
> 2025-07-03 at 7.05.42 PM-1.png, get_rule_violations_performance_tuning.patch
>
>
> Background: I have a very large flow with 50000 or so processors and the
> flow-analysis endpoint is taking 20 or more seconds to complete. This is
> likely because the endpoint is traversing all the process groups recursively
> each time it is called. While User A had the flow canvas open to the root
> node, User B was unable to make configuration changes to the flow and also
> had a very hard time making updates to the flow.
> This endpoint also holds a read lock on the flow for the entire time, so it
> will prevent any writes.
> Suggestions:
> # Short circuit this endpoint and return a default success value if there
> are no violations at all.
> # Only show detailed validation errors for the current process group and not
> descendants in the UI. For process groups within the current process group,
> show a cumulative count of violations within each process group. Counts would
> be updated lazily.
> # The UI only checks whether there exist violations at the current process
> group and descendants, only load the detailed validation errors if the user
> clicks on the first aid kit icon. A map of process groupID to boolean would
> be updated at the same time violations are updated and a violation on a
> process group would set the boolean to true for its ancestors up to the root.
> ## This can be tricky because process groups don't know their parent
> already. Also I think changing the meaning of the "first aid kit" to only
> correspond to the root PG, while it would help this performance situation, it
> would be less intuitive for the user if they're on any other process group
> than the root.
> # Keep a read through cache of PG id to violations. Empty it whenever the
> primary map of rule violations changes. On the validatior side, add
> exponential sleep between consecutive validation passes until the flow stops
> being modified.
> Screenshots from browser dev tools attached
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
