srowen commented on a change in pull request #23798: [SPARK-26890][DOC] Add list of available Dropwizard metrics in Spark and add additional configuration details to the monitoring documentation URL: https://github.com/apache/spark/pull/23798#discussion_r259372436
########## File path: docs/monitoring.md ########## @@ -684,9 +688,308 @@ code in your Spark package**_. For sbt users, set the the `-Pspark-ganglia-lgpl` profile. In addition to modifying the cluster's Spark build user applications will need to link to the `spark-ganglia-lgpl` artifact. -The syntax of the metrics configuration file is defined in an example configuration file, +The syntax of the metrics configuration file and the parameters available for each sink are defined +in an example configuration file, `$SPARK_HOME/conf/metrics.properties.template`. +When using configuration parameters instead of the metrics configuration file, the relevant +parameter names have the following form: +`spark.metrics.conf.[instance|*].sink.[sink_name].[parameter_name]`. +This example shows a list of configuration parameters for a Graphite sink: +``` +"spark.metrics.conf.*.sink.graphite.class"="org.apache.spark.metrics.sink.GraphiteSink" +"spark.metrics.conf.*.sink.graphite.host"="graphiteEndPoint_hostName>" +"spark.metrics.conf.*.sink.graphite.port"=<graphite_listening_port> +"spark.metrics.conf.*.sink.graphite.period"=10 +"spark.metrics.conf.*.sink.graphite.unit"=seconds +"spark.metrics.conf.*.sink.graphite.prefix"="optional_prefix" +``` + +Default values of the Spark metrics configuration are as follows: +``` +"*.sink.servlet.class" = "org.apache.spark.metrics.sink.MetricsServlet" +"*.sink.servlet.path" = "/metrics/json" +"master.sink.servlet.path" = "/metrics/master/json" +"applications.sink.servlet.path" = "/metrics/applications/json" +``` + +Additional sources can be configured using the metrics configuration file or the configuration +parameter `spark.metrics.conf.[component_name].source.jvm.class=[source_name]`. At present the + JVM source is the only available optional source. For example the following configuration parameter +activates the JVM source: +`"spark.metrics.conf.*.source.jvm.class"="org.apache.spark.metrics.source.JvmSource"` + +## List of available metrics providers + +Metrics used by Spark are of multiple types: gauge, counter, histogram, meter and timer, +see (Dropwizard libary documentation for details)[https://metrics.dropwizard.io/4.0.0/manual/core.html]. Review comment: I think we're on Dropwizard 3.1.5 right now though ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
