[
https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224453#comment-15224453
]
ASF subversion and git services commented on NIFI-1563:
-------------------------------------------------------
Commit 0d3bd2c401aacc5bcd602afe90883d751ff773f3 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0d3bd2c ]
NIFI-1563: - Federate requests and merge responses from nodes instead of
storing bulletins and stats at NCM - Updating UI to support restructured status
history DTO. - Return 'Insufficient History' message if aggregate stats don't
have enough data points, even if all nodes do (which can be the case if the
node performing the aggregation has a different value for the
'nifi.components.status.snapshot.frequency' property than the other nodes) -
Bug fixes; code cleanup; replicate requests to bulletin board endpoint -
Refactored the <Component>StatusDTO objects into <Component>StatusDTO,
<Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects -
Introducing endpoints for accessing individual component status. - Wiring up
new endpoints and updated core. - Code clean up. - Starting to handling status
merging of individual components. - Nodewise breakdown has been added to
Processors but the remaining components still need to be updated. - Refactor so
that System Diagnostics requests are replicated to nodes instead of the
information being pulled from Heartbeats - Replicate request for counters
instead of pulling them from heartbeats - Removed the getCounters / setCounters
method from HeartbeatPayload - Implementing component specific endpoints. -
Removing unused endpoints. - Supporting nodewise breakdown for system
diagnostics and counters. - Updating DTOs to use more consistent naming. - Code
clean up. - Addressing contrib issues. - Removed ProcessGroupStatus from
HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had
included it for testing that option but did not intend for it to be committed.
- Addressing comments in PR #294. - This closes #294
Signed-off-by: Matt Gilman <[email protected]>
> Remove storage of components' stats and bulletins from NCM
> ----------------------------------------------------------
>
> Key: NIFI-1563
> URL: https://issues.apache.org/jira/browse/NIFI-1563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 1.0.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the
> stats for all components in the node. This happens every 5 seconds by
> default. This results in quite a lot of chatter between the NCM and nodes. It
> made sense to take this approach when the clustering concept was designed
> because there were no Process Groups, and we had no notion of merging
> responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the
> responses on demand, rather than storing this information on the NCM. This
> requires far less bandwidth because we only need to pull the stats for a
> particular Process Group and only on demand instead of every 5 seconds.
> Additionally, this is laying groundwork for the Zero-Master clustering that
> we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the
> bulletins and stats history. These requests will all need to be federated and
> the responses merged on-demand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
