I have explored couple of observations and wanted to share seek necessary guidance
One cons i see with Shards is - if you have reached a limit on max targets (i.e total scrape time taking longer than intended scrape interval) or if you can’t vertically scale the pod/machine anymore. Sharding complicates your setup (you need more component to get a unified view), adds to query latency (due to fanout) and increases your costs. In that case - Do we have any approach/methodology of determining the “number of shards” based on the and number of targets/scrapping ? On Sunday, August 30, 2020 at 7:46:33 PM UTC+5:30 [email protected] wrote: > Can someone just share the knowledge about trades on sharding Prometheus > into multiple nodes > > Any hints/pointers would definitely be helpful > > On Fri, 28 Aug, 2020, 12:31 pm [email protected], < > [email protected]> wrote: > >> Hi Team, >> >> we have implemented huge number of shards in prometheus which scrapes >> 1000 target nodes across the enterprise >> >> we are currently in process of running some load test and ascertain the >> unknowns one below and need some guidance on the same - >> >> >> 1. On what cases sharding approach can fail? >> 2. How do we quantify the load/scale requirements? >> 3. Is there any optimized settings/configurations or best practices >> we need to comply for sharing >> 4. Any low level tuneable like recording >> rules/metrics_relabel_configs/scrape_internal and sample we ingest we >> need >> to accomodate and any pointers to it ? >> >> Please do guide us through whether we think on right page and correct me >> incase of any misinterpretation >> >> Regards, >> Dinesh >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/3307c7af-6fac-4e5a-a422-d2eee13c9fccn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/3307c7af-6fac-4e5a-a422-d2eee13c9fccn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/000f64ca-149b-48f1-8846-416182ae3f5en%40googlegroups.com.
