Hi Stuart, Thanks for the prompt response and all the guidance till date.
The set up we are looking for is that the user of the Grafana portal need not have access to absolutely any other piece of infrastructure (including the other Kubernetes clusters which are scraped for metrics). So what we have thought is to have all the Kubernetes clusters push their metrics to a Centralized Prometheus ... and have the Grafana sitting on top of only that Centralized Prometheus server. I was able to set-up the Prometheus server to server communication using Prometheus Federation as you have correctly suggested. However I am still reading for what metrics I may miss if I use the Prometheus Federation. In all I have the below three queries: 1. Are all the metrics forwarded using Prometheus Federation? Or is it that only a few are forwarded? 2. The metrics that are forwarded using Prometheus Federation, do they get stored in the TSDB of the destination Prometheus Server? 3. What would be the best way to take the back up of the Centralized Prometheus server? Do we need to use any external source like Thanos? Or are the disk backups of the Centralized Prometheus Server enough? Regards, Saurabh On Thu, Feb 25, 2021 at 2:23 PM Stuart Clark <[email protected]> wrote: > On 25/02/2021 06:55, Saurabh Vartak wrote: > > Hi Stuart, > > Thanks for all the help on this. I have created a Grafana cloud account > and their documentation states that it is possible to send the metrics from > one Prometheus server to the Prometheus instance on Grafana cloud: > https://grafana.com/docs/grafana-cloud/metrics/prometheus/ > > remote_write:- url: https://prometheus-us-central1.grafana.net/api/prom/push > basic_auth: > username: <Your Metrics instance ID> > password: <Your Grafana.com API Key> > > > I thought that this can be duplicated for my on-premise set up. I am > assuming that it is the Pushgateway to which the metrics are pushed from > the source Prometheus server to destination Prometheus server. Any ideas on > this? > > Grafana may be using the experimental feature I mentioned or other custom > code. > > Pushgateway is not useful here - it is designed for short lived processes > (such as cron jobs) which can't be scraped directly, and uses a different > API (not the remote write API). > > Maybe it would help if you could describe what you are trying to do from a > non-technical perspective? Why are you trying to send metrics from one > server to another? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKGwQmCHmJ5VGzPkT8wcup4hd8%3Dt6uYeXs5OB0--_qPphN-VLQ%40mail.gmail.com.
