[jira] [Commented] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags

2019-08-19 Thread ouyangwulin (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910926#comment-16910926
 ] 

ouyangwulin commented on FLINK-13418:
-

I use influxdbReporter in product env in NTES .When the flink task growing, the 
tags are increasing fast, and disk io of the influxdb meshine are not enough 
for the increasing index. As the temp method , I change to ssd. So, I think the 
Jira is a good idea.

> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acording to scope format) or 
> only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags

2019-08-16 Thread Yun Tang (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909034#comment-16909034
 ] 

Yun Tang commented on FLINK-13418:
--

[~Zentol], the comparison of these two different concepts is listed below and 
summarized from [influxdb official 
doc|https://docs.influxdata.com/influxdb/v1.7/concepts/key_concepts/#field-set]:
||concept||whether required||whether indexed||whether count for series||
|field|required|not indexed|no|
|tag|optional|indexed|yes|

 Take our {{numBytesInRemotePerSecond}} of input channel metrics as example, it 
is a task scope metrics with default naming 
{{.taskmanager}}. If we 
want to know the change of bytes input throughput of specific tasks, we need 
query with specific {{task_name}} and {{subtask_index}}. Just like 
[doc|https://docs.influxdata.com/influxdb/v1.7/concepts/key_concepts/#tag-set] 
said, if {{task_name}} and {{subtask_index}} were filed instead of tag, InfuxDB 
have to scan every value of {{task_name}} and {{subtask_index}}. On the other 
side, these could be set as tags to optimize the performance as tag is indexed 
in influxDB. The value of {{numBytesInRemotePerSecond}} is appropriate to act 
as field as we would not query on condign of values in most cases.

However, in InfluxDB, a {{series}} is the collection of data that share a 
retention policy, measurement, and tag set. In other words, more tags, more 
series could be counted. Since series number would impact the overall index 
performance especially the default in-memory index version for InfluxDB, and 
the default limit for series per data-base is only one million. If we include 
the {{task_attempt_id}} and other unnecessary tags in influxDB, the total 
series number would increase dramatically especially {{task_attempt_id}} set 
would be a high dimension set. That's why [~TheoD] come across the huge memory 
usage of InfluxDB.

 

> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acording to scope format) or 
> only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags

2019-08-16 Thread Chesnay Schepler (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908869#comment-16908869
 ] 

Chesnay Schepler commented on FLINK-13418:
--

Can someone provide me with a guide on best-practices as to what should be 
exposed as a field/tag? We can make this configurable, but If there are 
established best-practices I'd like to set up reasonable defaults.

> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acording to scope format) or 
> only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags

2019-08-14 Thread Yun Tang (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907169#comment-16907169
 ] 

Yun Tang commented on FLINK-13418:
--

[~TheoD] Before this issue is agreed to work (CC [~Zentol]), I think you could 
configure your influxDB with {{index-version="tsi1"}} in {{[data]}} session 
instead of default {{"inmem"}} index format to help decrease the usage of 
memory. 
BTW, If some specific user really want to store {{task_attempt_num}}, as 
influxDB would only be indexed with {{tags}}, they cannot act as {{field}}.

> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acording to scope format) or 
> only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13418) Avoid InfluxdbReporter to report unnecessary tags

2019-08-14 Thread Theo Diefenthal (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907102#comment-16907102
 ] 

Theo Diefenthal commented on FLINK-13418:
-

I see this issue as highly important for useage of InfluxDB reporter.

In our environment, we had a frequent crash of jobs which lead to 
"task_attempt_num" (which really is a number but reported as tag) going from 1 
to 1000 and thus breaking InfluxDb 1 million default value of series per 
database. After setting the limit to 50.000.000 and several more restarts of 
our tasks, we now observe that InfluxDB takes over 60GB of RAM for managing all 
the different series.

We really need a method here to
 * Ignore some variables/tags
 * Adjust some variables to be not stored as tags but as field (like 
task_attempt_num), which would also be better for querying that data.

> Avoid InfluxdbReporter to report unnecessary tags
> -
>
> Key: FLINK-13418
> URL: https://issues.apache.org/jira/browse/FLINK-13418
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Metrics
>Reporter: Yun Tang
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently, when building measurement info within {{InfluxdbReporter}}, it 
> would involve all variables as tags (please see code 
> [here|https://github.com/apache/flink/blob/d57741cef9d4773cc487418baa961254d0d47524/flink-metrics/flink-metrics-influxdb/src/main/java/org/apache/flink/metrics/influxdb/MeasurementInfoProvider.java#L54]).
>  However, user could adjust their own scope format to abort unnecessary 
> scope, while {{InfluxdbReporter}} could report all the scopes as tags to 
> InfluxDB.
> This is due to current {{MetricGroup}} lacks of any method to get necessary 
> scopes but only {{#getScopeComponents()}} or {{#getAllVariables()}}. In other 
> words, InfluxDB need tag-key and tag-value to compose as its tags while we 
> could only get all variables (without any filter acdording to scope format) 
> or only scopeComponents (could be treated as tag-value). I think that's why 
> previous implementation have to report all tags.
> From our experience on InfluxDB, as the size of tags contribute to the 
> overall series in InfluxDB, it would never be a good idea to contain too many 
> tags, not to mention the [default value of series per 
> database|https://docs.influxdata.com/influxdb/v1.7/troubleshooting/errors/#error-max-series-per-database-exceeded]
>  is only one million.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)