Theo Diefenthal commented on FLINK-13418:

I see this issue as highly important for useage of InfluxDB reporter.

In our environment, we had a frequent crash of jobs which lead to 
"task_attempt_num" (which really is a number but reported as tag) going from 1 
to 1000 and thus breaking InfluxDb 1 million default value of series per 
database. After setting the limit to 50.000.000 and several more restarts of 
our tasks, we now observe that InfluxDB takes over 60GB of RAM for managing all 
the different series.

We really need a method here to
 * Ignore some variables/tags
 * Adjust some variables to be not stored as tags but as field (like 
task_attempt_num), which would also be better for querying that data.

> Avoid InfluxdbReporter to report unnecessary tags
> -------------------------------------------------
>                 Key: FLINK-13418
>                 URL: https://issues.apache.org/jira/browse/FLINK-13418
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Metrics
>            Reporter: Yun Tang
>            Priority: Major
>             Fix For: 1.10.0
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acdording to scope format) 
> or only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.

This message was sent by Atlassian JIRA

Reply via email to