dthauvin commented on issue #40800:
URL: https://github.com/apache/airflow/issues/40800#issuecomment-2482650210
> > I think at the last call @ferruzzi mentioned that I was the proponent of
replacing StatsD with OpenTelemetry, but in fact it was a bit conditional
>
> My apologies for misrepresenting your stand on that. I shouldn't have
spoken for you.
>
> I have mentioned my solution in the past but perhaps I need to codify it,
or open an Issue and let someone else tackle it. Currently there are three
metrics backends (called loggers) StasD, OTel, and Datadog (which as far as I
can tell is a variant on StatsD which supports tagging). The biggest
compatibility issue is that OTel has a much shorter max name length (63
characters), but supports tagging. StatsD has a much longer (300 character?)
name length limit but does not support tagging.
>
> Solution:
>
> Add a "generate name" abstract method to base metrics logger, something
like:
>
> ```
> def get_name(name: str, tags:dict) -> str:
> ...
> ```
>
> Each logger will then need to implement this and call it before actually
emitting the metric. For otel, it would just return name. for StatsD it would
return the name with the tags dict concatenated onto it, something more or less
like this should work but there's likely a better way
>
> ```
> def get_name(name: str, tags:dict) -> str:
> suffix = str(tags)[1:-1] # drop the open and close curly braces
> .replace(": ", "_") # replace the key/value delimiter with an
underscore
> .replace(", ", "_") # replace the comma between keys/value pairs
with an underscore
> .replace("'", "") # remove the quotes around the key name
>
> return f"{name}_{suffix}"
> ```
>
> Then we can stop double-emitting, remove the "we have to truncate this
name" name length warning, and let each logger handle naming errors however it
sees fit.
>
> [EDIT] shorter but nowhere near as clear statsd implementation
>
> ```
> def get_name(name: str, tags:dict) -> str:
> return f'{name}_{("_").join([f"{k}_{v}" for (k, v) in tags.items()])}'
> ```
Hi @ferruzzi ,i don't know if you see this , but opentelemetry specification
goes from 63 characters to 255
https://github.com/open-telemetry/opentelemetry-specification/pull/3648 .
With OpenTelemetry configuration i got a lot of truncate metrics . Specially
those with the dag name inside . it's very difficult to predict the metrics
name and so make observability dashboard automation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
