IIRC the 'service.name' property in the metrics properties file is used to note from which service the metric was emitted. I would suggest testing this by setting the service.name property to tserver1 and see if your metrics show up with that name. If they do, then you might be able to modify the start scripts to have the different tserver instances use different metrics properties files.
From: Arvind Shyamsundar <[email protected]> Sent: Monday, August 26, 2019 8:39 PM To: [email protected] Subject: Uniquely identifying metrics with multiple tservers per node. Hello! We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally gathering metrics we are sending Accumulo and Hadoop metrics to collectd's statsd plugin (which acts as a statsd sink), which in turn forwards those metrics to an Azure monitoring service. The hadoop-metrics2-accumulo.properties looks like this: *.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink accumulo.sink.statsd.server.host=127.0.0.1 accumulo.sink.statsd.server.port=8125 accumulo.sink.statsd.skip.hostname=false # we have tried setting this to true and also to false, but that still does not include the service instance. accumulo.sink.statsd.service.name=master Everything works great when we have the default of 1 tserver per node. However, when running 2 tservers per node (Accumulo 1.9.3), we find that the metrics being reported (TabletServer.tserver.general.ingestRate for example) are not qualified / distinguished with the service instance ID (1_ or 2_) in the name. It appears as though only the values for one instance are being sent. As a result, we are unable to accurately measure aggregate metrics (as an example, the value of ingestRate recorded through this route is ~ 50% of what is shown on the Monitor UI). If someone has an example of how to configure hadoop-metrics2-accumulo.properties and multiple tservers per node in a way that the counter names would be distinguished by their service instance ID, that would be really helpful. Thank you! - Arvind.
