RE: Better way of measuring custom application metrics

[Shao, Saisai]

Hi,

I think there’s a StreamingSource in Spark Streaming which exposes the Spark 
Streaming running status to the metrics sink, you can connect it with Graphite 
sink to expose metrics to Graphite. I’m not sure is this what you want.

Besides you can customize the Source and Sink of the MetricsSystem to build 
your own and configure it in metrics.properties with class name to let it 
loaded by metrics system, for the details you can refer to 
http://spark.apache.org/docs/latest/monitoring.html or source code.

Thanks
Jerry

From: Enno Shioji [mailto:eshi...@gmail.com]
Sent: Sunday, January 4, 2015 7:47 AM
To: user@spark.apache.org
Subject: Better way of measuring custom application metrics

I have a hack to gather custom application metrics in a Streaming job, but I 
wanted to know if there is any better way of doing this.

My hack consists of this singleton:

object Metriker extends Serializable {
  @transient lazy val mr: MetricRegistry = {
    val metricRegistry = new MetricRegistry()
    val graphiteEndpoint = new 
InetSocketAddress("ec2-54-220-56-229.eu-west-1.compute.amazonaws.com<http://ec2-54-220-56-229.eu-west-1.compute.amazonaws.com>",
 2003)
    GraphiteReporter
      .forRegistry(metricRegistry)
      .build(new Graphite(graphiteEndpoint))
      .start(5, TimeUnit.SECONDS)
    metricRegistry
  }

  @transient lazy val processId = ManagementFactory.getRuntimeMXBean.getName

  @transient lazy val hostId = {
    try {
      InetAddress.getLocalHost.getHostName
    } catch {
      case e: UnknownHostException => "localhost"
    }
  }

   def metricName(name: String): String = {
    "%s.%s.%s".format(name, hostId, processId)
  }
}


which I then use in my jobs like so:

    stream
        .map { i =>
          Metriker.mr.meter(Metriker.metricName("testmetric123")).mark(i)
          i * 2
        }

Then I aggregate the metrics on Graphite. This works, but I was curious to know 
if anyone has a less hacky way.


[https://mailfoogae.appspot.com/t?sender=aZXNoaW9qaUBnbWFpbC5jb20%3D&type=zerocontent&guid=29916861-9b4d-423b-8e45-c731deddd43b]ᐧ