Thanks, Ben. One thing. I don't want to maintain 2 5TB disks for Prometheus HA, i.e 1 5TB disk on each instance, that is why I want to put a single huge disk in my VictoriaMetrics instance and maintain a single persistent disk rather than 2. Can Thanos also store the data in a persistent disk rather than Object disk? Because from the docs I have seen till now, I haven't found this feature in Thanos yet. This is the sole reason I am inclined towards VictoriaMetrics.
On Sunday, October 11, 2020 at 6:46:56 PM UTC+5:30 [email protected] wrote: > On Sun, Oct 11, 2020 at 10:45 AM [email protected] < > [email protected]> wrote: > >> On Saturday, October 10, 2020 at 2:32:14 PM UTC+5:30 [email protected] >> wrote: >> 4.6TB for 50 days seems like a lot. How many metrics and how many samples >> per second are you collecting? Just estimating based on the data, it sounds >> like you might have more than 10 million series and 600-700 samples per >> second. This might be the time to start thinking about sharding. >> You can check for sure with these queries: >> prometheus_tsdb_head_series >> rate(prometheus_tsdb_head_samples_appended_total[1h] >> >>>> >> Hi Ben, my time series collection hasn't touched 10 million yet, its >> around 5.5 million as of now, but my sampling rate is quite steep, sitting >> at approximately 643522. Since my time series are quite manageable by a >> single Prometheus instance I am avoiding sharding as of now because it >> would complicate the entire setup. What is your thought on this? >> > > I usually recommend thinking about a sharding plan when you hit this > level. You don't need to yet, but it's worth thinking about how you would. > > >> >> For handling HA clustering and sharding, I recommend looking into Thanos. >> It can be added to your existing Prometheus and rolled out incrementally. >> >>>> >> Yes, I looked at Thanos but my only problem is that Thanos will use >> Object Storage for long time retention which will have latency while >> extracting old data. That is why I am inclined towards VictoriaMetrics. >> What's your view on going with VictoriaMetrics? >> > > You don't need to use Thanos for long-term storage. It works just fine as > a query-proxy only setup. This is how we got into using Thanos. We had an > existing sharded fleet of Prometheus HA instances. We had been using > multiple Grafana data sources and simple nginx reverse proxy for HA > querying. We added Thanos Query/Sidecar just to provide a single query > interface. It wasn't until some time later that we started to use object > storage. > > Thanos object storage is optional, it can use Prometheus TSDB as the > backend. > > That said, Thanos object storage latency isn't a huge problem. It does > depend a bit on what object storage provider/software you use. But it works > just fine. > > I don't recommend VictoriaMetrics. I would go with Thanos or Cortex, as > these are maintained by core Prometheus community contributors. > > >> >> > d) d) If we do use 2 separate disks for the 2 instances, how will we >> manage the config files? >> If you don't have any configuration management, I recommend using >> https://github.com/cloudalchemy/ansible-prometheus. It's very easy to >> get going. >> >>>> >> Thanks. I'll check it out. >> >> -- >> > You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cb4928de-bb1f-4769-828b-6d9e10a3f532n%40googlegroups.com.
