[
https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208695#comment-15208695
]
ASF subversion and git services commented on NIFI-1563:
-------------------------------------------------------
Commit 137cccc868461860132097e01fdc32e583c33570 in nifi's branch refs/heads/1.x
from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=137cccc ]
NIFI-1563:
- Federate requests and merge responses from nodes instead of storing bulletins
and stats at NCM
- Updating UI to support restructured status history DTO.
- Return 'Insufficient History' message if aggregate stats don't have enough
data points, even if all nodes do (which can be the case if the node performing
the aggregation has a different value for the
'nifi.components.status.snapshot.frequency' property than the other nodes)
- Bug fixes; code cleanup; replicate requests to bulletin board endpoint
- Refactored the <Component>StatusDTO objects into <Component>StatusDTO,
<Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects
- Introducing endpoints for accessing individual component status.
- Wiring up new endpoints and updated core.
- Code clean up.
- Starting to handling status merging of individual components.
- Nodewise breakdown has been added to Processors but the remaining components
still need to be updated.
- Refactor so that System Diagnostics requests are replicated to nodes instead
of the information being pulled from Heartbeats
- Replicate request for counters instead of pulling them from heartbeats
- Removed the getCounters / setCounters method from HeartbeatPayload
- Implementing component specific endpoints.
- Removing unused endpoints.
- Supporting nodewise breakdown for system diagnostics and counters.
- Updating DTOs to use more consistent naming.
- Code clean up.
- Addressing contrib issues.
- Removed ProcessGroupStatus from HeartbeatPayload
- Removing nodewise from the system diagnostics endpoint. Had included it for
testing that option but did not intend for it to be committed.
- Addressing comments in PR #294.
- This closes #294
> Remove storage of components' stats and bulletins from NCM
> ----------------------------------------------------------
>
> Key: NIFI-1563
> URL: https://issues.apache.org/jira/browse/NIFI-1563
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 1.0.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the
> stats for all components in the node. This happens every 5 seconds by
> default. This results in quite a lot of chatter between the NCM and nodes. It
> made sense to take this approach when the clustering concept was designed
> because there were no Process Groups, and we had no notion of merging
> responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the
> responses on demand, rather than storing this information on the NCM. This
> requires far less bandwidth because we only need to pull the stats for a
> particular Process Group and only on demand instead of every 5 seconds.
> Additionally, this is laying groundwork for the Zero-Master clustering that
> we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the
> bulletins and stats history. These requests will all need to be federated and
> the responses merged on-demand.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
