1- How does it scale for thousands of agents?
With Ambari 2.5.2 you can launch a 2 node collector in distributed mode and select the placement of the node based on geographical location of the agents. Unfortunately, we shard based on hostname to select from the monitor, which collector to write to. Presently this is not customizable however, if you need it to be feel free to open a Jira, it is would be a straightforward fix required. However, I doubt you would need to do this because all sinks locally aggregate metrics and write only after 1 minute of data is collected. https://issues.apache.org/jira/browse/AMBARI-15901 Regarding volume of writes: We have tested the system with 2 node collector upto 2K nodes and we already know of production systems of up to 1500 node working with AMS. Since the collection model is push based the bottleneck needs to be solved on the collector and therefore in 2.5.1+ version of Ambari you can add additional collector node to handle extra load. Note: In 3.0 we are doing quite a few things to address performance by de-centralizing aggregations and reducing memory bloat on RegionServers etc, but that will be ready later in the year. 2- Can I use it over WAN probably by securing that in a two-way SSL way? Regarding security: We already support https for AMS - https://issues.apache.org/jira/browse/AMBARI-15050 The SPNEGO based authentication work is still not complete: We have a quite a few patches already in trunk but haven't looked at release readiness for these. Possible Qin Liu can shed more light on this. BR, Sid ________________________________ From: Ali Nazemian <[email protected]> Sent: Thursday, October 5, 2017 1:53 AM To: [email protected] Subject: Ambari metrics monitor-collector over the WAN Hi all, I am investigating to have a huge cluster of Nifi instances distributed across the world. I have been told that Metrics Monitor and Metrics Collector are using a very chatty protocol, so Ambari Metrics Collector may face some serious performance issues regarding receiving telemetry information from lots of Metrics Monitor Agents. I have two concerns regarding that as follows. 1- How does it scale for thousands of agents? 2- Can I use it over WAN probably by securing that in a two-way SSL way? Regards, Ali
