[
https://issues.apache.org/jira/browse/NIFI-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sheng Zhang updated NIFI-16570:
-------------------------------
Description:
Updating a Parameter Context that has a large number of parameters and is bound
to a large number of Process Groups takes tens of seconds, even when almost
nothing changes.
{code:java}
DtoFactory#createParameterContextDto{code}
builds the DTO as follows:
1. For every parameter it calls `createParameterEntity`.
2. `createParameterEntity` calls
`ParameterReferenceManager#getProcessorsReferencing` and
`#getControllerServicesReferencing` for that parameter.
3. In `StandardParameterReferenceManager`, each of those calls runs
`rootGroup.findAllProcessGroups(group ->
group.referencesParameterContext(parameterContext))`, i.e. a full walk of the
Process Group tree, and then scans the components of every matching group.
That is 2 tree walks per parameter, on top of one more walk in
`getProcessGroupsBound`. The cost is O(parameters x process groups).
The DTO is built several times during one update request:
- for the response of `PUT /parameter-contexts/\{id}`;
- when `ParameterUpdateManager` re-reads the context after replication;
- on every poll of `GET
/parameter-contexts/\{id}/update-requests/\{requestId}`.
h2. Measurements
Observed on a 2.5.0-based build, on a Parameter Context with 438 parameters
bound to 1426 Process Groups, total 90,000 processors in the cluster:
- About 18 ms per tree walk, so 438 x 2 x 18 ms is roughly 16 s per DTO build.
- The end-to-end update (PUT until the update request completes) took 33.9 s.
Measured numbers came from temporary timing logs on our fork. They are not part
of any patch.
h2. Proposed change
1. Add
`ParameterReferenceManager#getComponentsReferencingByParameterName(ParameterContext)`,
returning `Map<String, Set<ComponentNode>>`.
`StandardParameterReferenceManager` builds it in a single pass over the bound
Process Groups. A default implementation can return null so that other
implementations fall back to the current per-parameter lookup.
2. `DtoFactory#createParameterContextDto` builds the index once per call and
uses it for every parameter, instead of one lookup per parameter.
The output is identical to today and no API contract changes.
h3. Result (our fork)
On the same context, `createParameterContextDto` dropped from about 16 s to
42-122 ms, and the end-to-end update from 33.9 s to 0.92 s. These numbers were
measured on a build that also skipped building the 1426 bound Process Group
entities in the update response. That step takes about 69 ms per DTO build, so
it is not what makes the difference; the index is.
h2. Notes
I searched JIRA and apache/nifi issues and PRs and found nothing about this.
NIFI-7536 (timeouts while updating a Parameter Context) is related only in
symptom; it addressed the stop/replication wait, not DTO building.
I can open a PR with the index change and a unit test.
was:
Updating a Parameter Context that has a large number of parameters and is bound
to a large number of Process Groups takes tens of seconds, even when almost
nothing changes.
`DtoFactory#createParameterContextDto` builds the DTO as follows:
1. For every parameter it calls `createParameterEntity`.
2. `createParameterEntity` calls
`ParameterReferenceManager#getProcessorsReferencing` and
`#getControllerServicesReferencing` for that parameter.
3. In `StandardParameterReferenceManager`, each of those calls runs
`rootGroup.findAllProcessGroups(group ->
group.referencesParameterContext(parameterContext))`, i.e. a full walk of the
Process Group tree, and then scans the components of every matching group.
That is 2 tree walks per parameter, on top of one more walk in
`getProcessGroupsBound`. The cost is O(parameters x process groups).
The DTO is built several times during one update request:
- for the response of `PUT /parameter-contexts/\{id}`;
- when `ParameterUpdateManager` re-reads the context after replication;
- on every poll of `GET /parameter-contexts/\{id}/update-requests/\{requestId}`.
### Measurements
Observed on a 2.5.0-based build, on a Parameter Context with 438 parameters
bound to 1426 Process Groups:
- About 18 ms per tree walk, so 438 x 2 x 18 ms is roughly 16 s per DTO build.
- The end-to-end update (PUT until the update request completes) took 33.9 s.
Measured numbers came from temporary timing logs on our fork. They are not part
of any patch.
### Proposed change
1. Add
`ParameterReferenceManager#getComponentsReferencingByParameterName(ParameterContext)`,
returning `Map<String, Set<ComponentNode>>`.
`StandardParameterReferenceManager` builds it in a single pass over the bound
Process Groups. A default implementation can return null so that other
implementations fall back to the current per-parameter lookup.
2. `DtoFactory#createParameterContextDto` builds the index once per call and
uses it for every parameter, instead of one lookup per parameter.
The output is identical to today and no API contract changes.
### Result (our fork)
On the same context, `createParameterContextDto` dropped from about 16 s to
42-122 ms, and the end-to-end update from 33.9 s to 0.92 s. These numbers were
measured on a build that also skipped building the 1426 bound Process Group
entities in the update response. That step takes about 69 ms per DTO build, so
it is not what makes the difference; the index is.
## Notes
I searched JIRA and apache/nifi issues and PRs and found nothing about this.
NIFI-7536 (timeouts while updating a Parameter Context) is related only in
symptom; it addressed the stop/replication wait, not DTO building.
I can open a PR with the index change and a unit test.
> Updating a Parameter Context with many parameters and many bound Process
> Groups is slow: building the Parameter Context DTO walks the whole Process
> Group tree twice per parameter
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16570
> URL: https://issues.apache.org/jira/browse/NIFI-16570
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 2.5.0
> Reporter: Sheng Zhang
> Priority: Major
>
> Updating a Parameter Context that has a large number of parameters and is
> bound to a large number of Process Groups takes tens of seconds, even when
> almost nothing changes.
>
> {code:java}
> DtoFactory#createParameterContextDto{code}
> builds the DTO as follows:
>
> 1. For every parameter it calls `createParameterEntity`.
> 2. `createParameterEntity` calls
> `ParameterReferenceManager#getProcessorsReferencing` and
> `#getControllerServicesReferencing` for that parameter.
> 3. In `StandardParameterReferenceManager`, each of those calls runs
> `rootGroup.findAllProcessGroups(group ->
> group.referencesParameterContext(parameterContext))`, i.e. a full walk of the
> Process Group tree, and then scans the components of every matching group.
> That is 2 tree walks per parameter, on top of one more walk in
> `getProcessGroupsBound`. The cost is O(parameters x process groups).
> The DTO is built several times during one update request:
> - for the response of `PUT /parameter-contexts/\{id}`;
> - when `ParameterUpdateManager` re-reads the context after replication;
> - on every poll of `GET
> /parameter-contexts/\{id}/update-requests/\{requestId}`.
> h2. Measurements
> Observed on a 2.5.0-based build, on a Parameter Context with 438 parameters
> bound to 1426 Process Groups, total 90,000 processors in the cluster:
> - About 18 ms per tree walk, so 438 x 2 x 18 ms is roughly 16 s per DTO
> build.
> - The end-to-end update (PUT until the update request completes) took 33.9 s.
> Measured numbers came from temporary timing logs on our fork. They are not
> part of any patch.
> h2. Proposed change
> 1. Add
> `ParameterReferenceManager#getComponentsReferencingByParameterName(ParameterContext)`,
> returning `Map<String, Set<ComponentNode>>`.
> `StandardParameterReferenceManager` builds it in a single pass over the bound
> Process Groups. A default implementation can return null so that other
> implementations fall back to the current per-parameter lookup.
> 2. `DtoFactory#createParameterContextDto` builds the index once per call and
> uses it for every parameter, instead of one lookup per parameter.
> The output is identical to today and no API contract changes.
> h3. Result (our fork)
> On the same context, `createParameterContextDto` dropped from about 16 s to
> 42-122 ms, and the end-to-end update from 33.9 s to 0.92 s. These numbers
> were measured on a build that also skipped building the 1426 bound Process
> Group entities in the update response. That step takes about 69 ms per DTO
> build, so it is not what makes the difference; the index is.
> h2. Notes
> I searched JIRA and apache/nifi issues and PRs and found nothing about this.
> NIFI-7536 (timeouts while updating a Parameter Context) is related only in
> symptom; it addressed the stop/replication wait, not DTO building.
> I can open a PR with the index change and a unit test.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
