Dropping that label would only work if all instances of that metric have
the same value for that label - in which case, there wasn't a cardinality
problem in the first place. If there are distinct values, then dropping it
will most likely end up with duplicate instances of the same timeseries,
which will be rejected.
The only way I can think of working around this is to write some
intermediate proxy which collects all the metrics, sums across the
appropriate dimension(s), and exposes the results as a new metric.
But really, the answer is to push back. You are getting junk metrics which
cause the TSDB to explode; therefore either the metrics need to be fixed,
or you simply stop collecting them.
On Friday, 21 January 2022 at 15:54:58 UTC Stuart Clark wrote:
> On 21/01/2022 14:09, Ramachandra Bhaskar Ayalavarapu wrote:
>
> We have some data points like this
> {
> "__name__": "http_request_duration_seconds_bucket",
> "endpoint": "apssd",
> "instance": "10.10.77.218:8080",
> "job": "api-service",
> "le": "0.05",
> "method": "GET",
> "namespace": "foo",
> "pod": "bar",
> "route": "/localhsdssds/v1/project/<unique
> projectid>/provider/WmRESTProvider/blasdsdsd/fummssd/icon",
> "service": "api-service",
> "status": "4XX"
> }
>
>
> unfortunately though we understand its not best practice to add "route" in
> the label name which is driving cardinality we cannot do anything as team
> is not ready to change code. Can i put a first aid any way to reduce the
> cardinality. (this metric) is too important to drop
>
> some thing like project wise etc by writing rewrite rule. if yes please
> advice how ?
>
> You could drop that label in metrics relabling.
>
> --
> Stuart Clark
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/10f9a030-124e-48e7-9654-e336c9f55f7bn%40googlegroups.com.
