Thanks, Ben. What happens to Alerting in case of HA Prometheus while using Thanos/VictoriaMetrics/Cortex on top of 2 Prometheus instances?
On Monday, October 12, 2020 at 12:57:17 AM UTC+5:30 [email protected] wrote: > I don't think this is something you can or should be optimizing for. You > are on the edge of needing to shard, which means you will need to manage > many individual instance disks. > > But if you really want to have a single big disk for storage, you can use > Minio[0] for simple object storage if you aren't already on a platform that > provides object storage. > > [0]: https://min.io/ > > On Sun, Oct 11, 2020, 18:29 [email protected] <[email protected]> > wrote: > >> Thanks, Ben. >> >> One thing. I don't want to maintain 2 5TB disks for Prometheus HA, i.e 1 >> 5TB disk on each instance, that is why I want to put a single huge disk in >> my VictoriaMetrics instance and maintain a single persistent disk rather >> than 2. Can Thanos also store the data in a persistent disk rather than >> Object disk? Because from the docs I have seen till now, I haven't found >> this feature in Thanos yet. This is the sole reason I am inclined towards >> VictoriaMetrics. >> >> On Sunday, October 11, 2020 at 6:46:56 PM UTC+5:30 [email protected] >> wrote: >> >>> On Sun, Oct 11, 2020 at 10:45 AM [email protected] < >>> [email protected]> wrote: >>> >>>> On Saturday, October 10, 2020 at 2:32:14 PM UTC+5:30 [email protected] >>>> wrote: >>>> 4.6TB for 50 days seems like a lot. How many metrics and how many >>>> samples per second are you collecting? Just estimating based on the data, >>>> it sounds like you might have more than 10 million series and 600-700 >>>> samples per second. This might be the time to start thinking about >>>> sharding. >>>> You can check for sure with these queries: >>>> prometheus_tsdb_head_series >>>> rate(prometheus_tsdb_head_samples_appended_total[1h] >>>> >>>> >>>> Hi Ben, my time series collection hasn't touched 10 million yet, its >>>> around 5.5 million as of now, but my sampling rate is quite steep, sitting >>>> at approximately 643522. Since my time series are quite manageable by a >>>> single Prometheus instance I am avoiding sharding as of now because it >>>> would complicate the entire setup. What is your thought on this? >>>> >>> >>> I usually recommend thinking about a sharding plan when you hit this >>> level. You don't need to yet, but it's worth thinking about how you would. >>> >>> >>>> >>>> For handling HA clustering and sharding, I recommend looking into >>>> Thanos. It can be added to your existing Prometheus and rolled out >>>> incrementally. >>>> >>>> >>>> Yes, I looked at Thanos but my only problem is that Thanos will use >>>> Object Storage for long time retention which will have latency while >>>> extracting old data. That is why I am inclined towards VictoriaMetrics. >>>> What's your view on going with VictoriaMetrics? >>>> >>> >>> You don't need to use Thanos for long-term storage. It works just fine >>> as a query-proxy only setup. This is how we got into using Thanos. We had >>> an existing sharded fleet of Prometheus HA instances. We had been using >>> multiple Grafana data sources and simple nginx reverse proxy for HA >>> querying. We added Thanos Query/Sidecar just to provide a single query >>> interface. It wasn't until some time later that we started to use object >>> storage. >>> >>> Thanos object storage is optional, it can use Prometheus TSDB as the >>> backend. >>> >>> That said, Thanos object storage latency isn't a huge problem. It does >>> depend a bit on what object storage provider/software you use. But it works >>> just fine. >>> >>> I don't recommend VictoriaMetrics. I would go with Thanos or Cortex, as >>> these are maintained by core Prometheus community contributors. >>> >>> >>>> >>>> > d) d) If we do use 2 separate disks for the 2 instances, how will we >>>> manage the config files? >>>> If you don't have any configuration management, I recommend using >>>> https://github.com/cloudalchemy/ansible-prometheus. It's very easy to >>>> get going. >>>> >>>> >>>> Thanks. I'll check it out. >>>> >>>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/cb4928de-bb1f-4769-828b-6d9e10a3f532n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/cb4928de-bb1f-4769-828b-6d9e10a3f532n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f2dc0f42-8e7c-4a26-9dc0-94ecefb32e19n%40googlegroups.com.
