rmannibucau commented on PR #3385: URL: https://github.com/apache/polaris/pull/3385#issuecomment-3856157729
@obelix74 personally I'd like it to be unified in the monitoring stack so stored through the opentelemetry-collector and not at all in a custom manner. If using tempo it will end up in parquet files on s3 for ex, if using openobserve in almost direct iceberg tables, if using opentelemetry-collector + kafka + kafka connect it can be in iceberg, but at the end the storage is fully managed by the observability stack so no issue in terms of cleanup. Using a cron with ttl can be challenging since a lot of observability storage also take into account the size to drop data so you might easily end up not being able to do the join and having to implement a custom join in grafana can not be great since it will not work in explore mode - likely where you'll need these - but only in dashboard with drilldown capabilities. So from an user perspective I would 100% rely on span attributes (+ resource attribute for global ones) + `/metrics` and be it - doesn't change at all stored data. Side note: indeed it applies for the observability side of things, for _internal_ metrics (aka database statistics), it should just be next to puffin files I think - if there is any overlap it should be duplicated is what I mean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
