[
https://issues.apache.org/jira/browse/FLINK-16839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muchl updated FLINK-16839:
--------------------------
Description:
Currently there are 9 metrics in the kinesis connector, each of which is
recorded according to the kinesis shard dimension. If there are enough shards,
taskmanager mtrics will be unavailable.
In our production environment, a case of a job that reads dynamodb stream
kinesis adapter, this dynamodb has more than 10,000 shards, multiplied by 9
metrics, there are more than 100,000 metrics for kinesis, and the entire
metrics output reaches tens of MB , Cause prometheus cannot collect metrics.
We should have a configuration item that can disable kinesis metrics
was:
Currently, there are 9 metrics in the kinesis connector, each of which is
recorded according to the kinesis shard dimension. If there are enough shards,
taskmanager mtrics will be unavailable.
In our production environment, a case of a job that reads dynamodb stream
kinesis adapter, this dynamodb has more than 10,000 shards, multiplied by 9
metrics, there are more than 100,000 metrics for kinesis, and the entire
metrics output reaches tens of MB , Cause prometheus cannot collect metrics.
We should have a configuration item that can disable kinesis metrics
> Connectors Kinesis metrics can be disabled
> ------------------------------------------
>
> Key: FLINK-16839
> URL: https://issues.apache.org/jira/browse/FLINK-16839
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis, Runtime / Metrics
> Affects Versions: 1.10.0
> Reporter: Muchl
> Priority: Minor
>
> Currently there are 9 metrics in the kinesis connector, each of which is
> recorded according to the kinesis shard dimension. If there are enough
> shards, taskmanager mtrics will be unavailable.
> In our production environment, a case of a job that reads dynamodb stream
> kinesis adapter, this dynamodb has more than 10,000 shards, multiplied by 9
> metrics, there are more than 100,000 metrics for kinesis, and the entire
> metrics output reaches tens of MB , Cause prometheus cannot collect metrics.
> We should have a configuration item that can disable kinesis metrics
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
