I don't think this is something you can or should be optimizing for. You are on the edge of needing to shard, which means you will need to manage many individual instance disks.
But if you really want to have a single big disk for storage, you can use Minio[0] for simple object storage if you aren't already on a platform that provides object storage. [0]: https://min.io/ On Sun, Oct 11, 2020, 18:29 [email protected] <[email protected]> wrote: > Thanks, Ben. > > One thing. I don't want to maintain 2 5TB disks for Prometheus HA, i.e 1 > 5TB disk on each instance, that is why I want to put a single huge disk in > my VictoriaMetrics instance and maintain a single persistent disk rather > than 2. Can Thanos also store the data in a persistent disk rather than > Object disk? Because from the docs I have seen till now, I haven't found > this feature in Thanos yet. This is the sole reason I am inclined towards > VictoriaMetrics. > > On Sunday, October 11, 2020 at 6:46:56 PM UTC+5:30 [email protected] wrote: > >> On Sun, Oct 11, 2020 at 10:45 AM [email protected] < >> [email protected]> wrote: >> >>> On Saturday, October 10, 2020 at 2:32:14 PM UTC+5:30 [email protected] >>> wrote: >>> 4.6TB for 50 days seems like a lot. How many metrics and how many >>> samples per second are you collecting? Just estimating based on the data, >>> it sounds like you might have more than 10 million series and 600-700 >>> samples per second. This might be the time to start thinking about sharding. >>> You can check for sure with these queries: >>> prometheus_tsdb_head_series >>> rate(prometheus_tsdb_head_samples_appended_total[1h] >>> >>>> >>> Hi Ben, my time series collection hasn't touched 10 million yet, its >>> around 5.5 million as of now, but my sampling rate is quite steep, sitting >>> at approximately 643522. Since my time series are quite manageable by a >>> single Prometheus instance I am avoiding sharding as of now because it >>> would complicate the entire setup. What is your thought on this? >>> >> >> I usually recommend thinking about a sharding plan when you hit this >> level. You don't need to yet, but it's worth thinking about how you would. >> >> >>> >>> For handling HA clustering and sharding, I recommend looking into >>> Thanos. It can be added to your existing Prometheus and rolled out >>> incrementally. >>> >>>> >>> Yes, I looked at Thanos but my only problem is that Thanos will use >>> Object Storage for long time retention which will have latency while >>> extracting old data. That is why I am inclined towards VictoriaMetrics. >>> What's your view on going with VictoriaMetrics? >>> >> >> You don't need to use Thanos for long-term storage. It works just fine as >> a query-proxy only setup. This is how we got into using Thanos. We had an >> existing sharded fleet of Prometheus HA instances. We had been using >> multiple Grafana data sources and simple nginx reverse proxy for HA >> querying. We added Thanos Query/Sidecar just to provide a single query >> interface. It wasn't until some time later that we started to use object >> storage. >> >> Thanos object storage is optional, it can use Prometheus TSDB as the >> backend. >> >> That said, Thanos object storage latency isn't a huge problem. It does >> depend a bit on what object storage provider/software you use. But it works >> just fine. >> >> I don't recommend VictoriaMetrics. I would go with Thanos or Cortex, as >> these are maintained by core Prometheus community contributors. >> >> >>> >>> > d) d) If we do use 2 separate disks for the 2 instances, how will we >>> manage the config files? >>> If you don't have any configuration management, I recommend using >>> https://github.com/cloudalchemy/ansible-prometheus. It's very easy to >>> get going. >>> >>>> >>> Thanks. I'll check it out. >>> >>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/6cfb736c-ba38-4e8d-8468-cdc84f2971f2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/cb4928de-bb1f-4769-828b-6d9e10a3f532n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/cb4928de-bb1f-4769-828b-6d9e10a3f532n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo1Jf%3DcKHtvoZH_%3DpcroqOGBYu_OgHUCyaPcH72XAbXJQ%40mail.gmail.com.
