This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new f2b7486 [FLINK-26886][metrics][docs] Document pull/push
tag/identifier reporter behavior
f2b7486 is described below
commit f2b748665d0682b92c5d72653687a201ee7abbfd
Author: Chesnay Schepler <[email protected]>
AuthorDate: Fri Mar 25 09:32:13 2022 +0100
[FLINK-26886][metrics][docs] Document pull/push tag/identifier reporter
behavior
---
.../content.zh/docs/deployment/metric_reporters.md | 39 ++++++++++++++++++++--
docs/content/docs/deployment/metric_reporters.md | 39 ++++++++++++++++++++--
.../apache/flink/configuration/MetricOptions.java | 19 ++++++-----
3 files changed, 82 insertions(+), 15 deletions(-)
diff --git a/docs/content.zh/docs/deployment/metric_reporters.md
b/docs/content.zh/docs/deployment/metric_reporters.md
index 7c432e5..d65970b 100644
--- a/docs/content.zh/docs/deployment/metric_reporters.md
+++ b/docs/content.zh/docs/deployment/metric_reporters.md
@@ -29,9 +29,6 @@ under the License.
Flink allows reporting metrics to external systems.
For more information about Flink's metric system go to the [metric system
documentation]({{< ref "docs/ops/metrics" >}}).
-
-## Reporter
-
Metrics can be exposed to an external system by configuring one or several
reporters in `conf/flink-conf.yaml`. These
reporters will be instantiated on each job and task manager when they are
started.
@@ -66,11 +63,33 @@ You can write your own `Reporter` by implementing the
`org.apache.flink.metrics.
If the Reporter should send out reports regularly you have to implement the
`Scheduled` interface as well.
By additionally implementing a `MetricReporterFactory` your reporter can also
be loaded as a plugin.
+## Identifiers vs. tags
+
+There are generally 2 formats for how reporters export metrics.
+
+Identifier-based reporters assemble a flat string containing all scope
information and the metric name.
+An example could be `job.MyJobName.numRestarts`.
+
+Tag-based reporters on the other hand define a generic class of metrics
consisting of a logical scope and metric name (e.g., `job.numRestarts`),
+and report a particular instance of said metric as a set of `key-value` pairs,
so called "tags" or "variables" (e.g., "jobName=MyJobName").
+
+## Push vs. Pull
+
+Metrics are exported either via pushes or pulls.
+
+Push-based reporters usually implement the `Scheduled` interface and
periodically send a summary of current metrics to an external system.
+
+Pull-based reporters are queried from an external system instead.
+
+## Reporters
+
The following sections list the supported reporters.
### JMX
#### (org.apache.flink.metrics.jmx.JMXReporter)
+Type: pull/tags
+
Parameters:
- `port` - (optional) the port on which JMX listens for connections.
@@ -101,6 +120,8 @@ The domain thus identifies a metric class, while the
key-property list identifie
### Graphite
#### (org.apache.flink.metrics.graphite.GraphiteReporter)
+Type: push/identifier
+
Parameters:
- `host` - the Graphite server host
@@ -120,6 +141,8 @@ metrics.reporter.grph.interval: 60 SECONDS
### InfluxDB
#### (org.apache.flink.metrics.influxdb.InfluxdbReporter)
+Type: push/tags
+
Parameters:
{{< include_reporter_config
"layouts/shortcodes/generated/influxdb_reporter_configuration.html" >}}
@@ -147,6 +170,8 @@ All Flink metrics variables (see [List of all
Variables]({{< ref "docs/ops/metri
### Prometheus
#### (org.apache.flink.metrics.prometheus.PrometheusReporter)
+Type: pull/tags
+
Parameters:
- `port` - (optional) the port the Prometheus exporter listens on, defaults to
[9249](https://github.com/prometheus/prometheus/wiki/Default-port-allocations).
In order to be able to run several instances of the reporter on one host (e.g.
when one TaskManager is colocated with the JobManager) it is advisable to use a
port range like `9250-9260`.
@@ -172,6 +197,8 @@ All Flink metrics variables (see [List of all
Variables]({{< ref "docs/ops/metri
### PrometheusPushGateway
#### (org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter)
+Type: push/tags
+
Parameters:
{{< include_reporter_config
"layouts/shortcodes/generated/prometheus_push_gateway_reporter_configuration.html"
>}}
@@ -195,6 +222,8 @@ Please see the [Prometheus
documentation](https://prometheus.io/docs/practices/p
### StatsD
#### (org.apache.flink.metrics.statsd.StatsDReporter)
+Type: push/identifier
+
Parameters:
- `host` - the StatsD server host
@@ -212,6 +241,8 @@ metrics.reporter.stsd.interval: 60 SECONDS
### Datadog
#### (org.apache.flink.metrics.datadog.DatadogHttpReporter)
+Type: push/tags
+
Note any variables in Flink metrics, such as `<host>`, `<job_name>`,
`<tm_id>`, `<subtask_index>`, `<task_name>`, and `<operator_name>`,
will be sent to Datadog as tags. Tags will look like `host:localhost` and
`job_name:myjobname`.
@@ -245,6 +276,8 @@ metrics.reporter.dghttp.interval: 60 SECONDS
### Slf4j
#### (org.apache.flink.metrics.slf4j.Slf4jReporter)
+Type: push/identifier
+
Example configuration:
```yaml
diff --git a/docs/content/docs/deployment/metric_reporters.md
b/docs/content/docs/deployment/metric_reporters.md
index c327b76..0f2eb25 100644
--- a/docs/content/docs/deployment/metric_reporters.md
+++ b/docs/content/docs/deployment/metric_reporters.md
@@ -29,9 +29,6 @@ under the License.
Flink allows reporting metrics to external systems.
For more information about Flink's metric system go to the [metric system
documentation]({{< ref "docs/ops/metrics" >}}).
-
-## Reporter
-
Metrics can be exposed to an external system by configuring one or several
reporters in `conf/flink-conf.yaml`. These
reporters will be instantiated on each job and task manager when they are
started.
@@ -66,11 +63,33 @@ You can write your own `Reporter` by implementing the
`org.apache.flink.metrics.
If the Reporter should send out reports regularly you have to implement the
`Scheduled` interface as well.
By additionally implementing a `MetricReporterFactory` your reporter can also
be loaded as a plugin.
+## Identifiers vs. tags
+
+There are generally 2 formats for how reporters export metrics.
+
+Identifier-based reporters assemble a flat string containing all scope
information and the metric name.
+An example could be `job.MyJobName.numRestarts`.
+
+Tag-based reporters on the other hand define a generic class of metrics
consisting of a logical scope and metric name (e.g., `job.numRestarts`),
+and report a particular instance of said metric as a set of `key-value` pairs,
so called "tags" or "variables" (e.g., "jobName=MyJobName").
+
+## Push vs. Pull
+
+Metrics are exported either via pushes or pulls.
+
+Push-based reporters usually implement the `Scheduled` interface and
periodically send a summary of current metrics to an external system.
+
+Pull-based reporters are queried from an external system instead.
+
+## Reporters
+
The following sections list the supported reporters.
### JMX
#### (org.apache.flink.metrics.jmx.JMXReporter)
+Type: pull/tags
+
Parameters:
- `port` - (optional) the port on which JMX listens for connections.
@@ -101,6 +120,8 @@ The domain thus identifies a metric class, while the
key-property list identifie
### Graphite
#### (org.apache.flink.metrics.graphite.GraphiteReporter)
+Type: push/identifier
+
Parameters:
- `host` - the Graphite server host
@@ -120,6 +141,8 @@ metrics.reporter.grph.interval: 60 SECONDS
### InfluxDB
#### (org.apache.flink.metrics.influxdb.InfluxdbReporter)
+Type: push/tags
+
Parameters:
{{< include_reporter_config
"layouts/shortcodes/generated/influxdb_reporter_configuration.html" >}}
@@ -147,6 +170,8 @@ All Flink metrics variables (see [List of all
Variables]({{< ref "docs/ops/metri
### Prometheus
#### (org.apache.flink.metrics.prometheus.PrometheusReporter)
+Type: pull/tags
+
Parameters:
- `port` - (optional) the port the Prometheus exporter listens on, defaults to
[9249](https://github.com/prometheus/prometheus/wiki/Default-port-allocations).
In order to be able to run several instances of the reporter on one host (e.g.
when one TaskManager is colocated with the JobManager) it is advisable to use a
port range like `9250-9260`.
@@ -172,6 +197,8 @@ All Flink metrics variables (see [List of all
Variables]({{< ref "docs/ops/metri
### PrometheusPushGateway
#### (org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter)
+Type: push/tags
+
Parameters:
{{< include_reporter_config
"layouts/shortcodes/generated/prometheus_push_gateway_reporter_configuration.html"
>}}
@@ -195,6 +222,8 @@ Please see the [Prometheus
documentation](https://prometheus.io/docs/practices/p
### StatsD
#### (org.apache.flink.metrics.statsd.StatsDReporter)
+Type: push/identifier
+
Parameters:
- `host` - the StatsD server host
@@ -212,6 +241,8 @@ metrics.reporter.stsd.interval: 60 SECONDS
### Datadog
#### (org.apache.flink.metrics.datadog.DatadogHttpReporter)
+Type: push/tags
+
Note any variables in Flink metrics, such as `<host>`, `<job_name>`,
`<tm_id>`, `<subtask_index>`, `<task_name>`, and `<operator_name>`,
will be sent to Datadog as tags. Tags will look like `host:localhost` and
`job_name:myjobname`.
@@ -245,6 +276,8 @@ metrics.reporter.dghttp.interval: 60 SECONDS
### Slf4j
#### (org.apache.flink.metrics.slf4j.Slf4jReporter)
+Type: push/identifier
+
Example configuration:
```yaml
diff --git
a/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java
b/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java
index 12ab390..c164e7e 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java
@@ -90,7 +90,8 @@ public class MetricOptions {
key("interval")
.durationType()
.defaultValue(Duration.ofSeconds(10))
- .withDescription("The reporter interval to use for the
reporter named <name>.");
+ .withDescription(
+ "The reporter interval to use for the reporter
named <name>. Only applicable to push-based reporters.");
@Documentation.SuffixOption(NAMED_REPORTER_CONFIG_PREFIX)
@Documentation.Section(value = Documentation.Sections.METRIC_REPORTERS,
position = 2)
@@ -108,7 +109,7 @@ public class MetricOptions {
.mapType()
.defaultValue(Collections.emptyMap())
.withDescription(
- "The map of additional variables that should be
included for the reporter named <name>. Only applicable to tag-based reporters
(e.g., PRometheus, InfluxDB).");
+ "The map of additional variables that should be
included for the reporter named <name>. Only applicable to tag-based
reporters.");
@Documentation.SuffixOption(NAMED_REPORTER_CONFIG_PREFIX)
@Documentation.Section(value = Documentation.Sections.METRIC_REPORTERS,
position = 3)
@@ -117,7 +118,7 @@ public class MetricOptions {
.stringType()
.defaultValue(".")
.withDescription(
- "The set of variables that should be excluded for
the reporter named <name>. Only applicable to tag-based reporters (e.g.,
PRometheus, InfluxDB).");
+ "The set of variables that should be excluded for
the reporter named <name>. Only applicable to tag-based reporters.");
@Documentation.SuffixOption(NAMED_REPORTER_CONFIG_PREFIX)
@Documentation.Section(value = Documentation.Sections.METRIC_REPORTERS,
position = 4)
@@ -230,7 +231,7 @@ public class MetricOptions {
.stringType()
.defaultValue("<host>.jobmanager")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to a JobManager.");
+ "Defines the scope format string that is applied
to all metrics scoped to a JobManager. Only effective when a identifier-based
reporter is configured.");
/** The scope format string that is applied to all metrics scoped to a
TaskManager. */
public static final ConfigOption<String> SCOPE_NAMING_TM =
@@ -238,7 +239,7 @@ public class MetricOptions {
.stringType()
.defaultValue("<host>.taskmanager.<tm_id>")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to a TaskManager.");
+ "Defines the scope format string that is applied
to all metrics scoped to a TaskManager. Only effective when a identifier-based
reporter is configured");
/** The scope format string that is applied to all metrics scoped to a job
on a JobManager. */
public static final ConfigOption<String> SCOPE_NAMING_JM_JOB =
@@ -246,7 +247,7 @@ public class MetricOptions {
.stringType()
.defaultValue("<host>.jobmanager.<job_name>")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to a job on a JobManager.");
+ "Defines the scope format string that is applied
to all metrics scoped to a job on a JobManager. Only effective when a
identifier-based reporter is configured");
/** The scope format string that is applied to all metrics scoped to a job
on a TaskManager. */
public static final ConfigOption<String> SCOPE_NAMING_TM_JOB =
@@ -254,7 +255,7 @@ public class MetricOptions {
.stringType()
.defaultValue("<host>.taskmanager.<tm_id>.<job_name>")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to a job on a TaskManager.");
+ "Defines the scope format string that is applied
to all metrics scoped to a job on a TaskManager. Only effective when a
identifier-based reporter is configured");
/** The scope format string that is applied to all metrics scoped to a
task. */
public static final ConfigOption<String> SCOPE_NAMING_TASK =
@@ -263,7 +264,7 @@ public class MetricOptions {
.defaultValue(
"<host>.taskmanager.<tm_id>.<job_name>.<task_name>.<subtask_index>")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to a task.");
+ "Defines the scope format string that is applied
to all metrics scoped to a task. Only effective when a identifier-based
reporter is configured");
/** The scope format string that is applied to all metrics scoped to an
operator. */
public static final ConfigOption<String> SCOPE_NAMING_OPERATOR =
@@ -272,7 +273,7 @@ public class MetricOptions {
.defaultValue(
"<host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>")
.withDescription(
- "Defines the scope format string that is applied
to all metrics scoped to an operator.");
+ "Defines the scope format string that is applied
to all metrics scoped to an operator. Only effective when a identifier-based
reporter is configured");
public static final ConfigOption<Long> LATENCY_INTERVAL =
key("metrics.latency.interval")
