sudo sysctl fs.inotify.max_user_watches=1048576 I ran the above command and it started working fine. But now my only concern is I'll have 2k(containers on each host) * 20(host). I have done relabeling already. Would this problem reproduce if I increase no. of hosts? Currently it is 1 host with 2k containers.
On Wednesday, April 29, 2020 at 11:21:23 AM UTC+5:30, Isabel Noronha wrote: > > Yes I changed it 40 s. > I had increased it since in one host 2k containers were running and then > there was this error "context deadline exceeded error". > So I thought scrape_interval isn't sufficient to scrape all the metrics. > > > > On Wednesday, April 29, 2020 at 2:38:58 AM UTC+5:30, Stuart Clark wrote: >> >> On 28/04/2020 20:31, Isabel Noronha wrote: >> > scrape_interval is 5m >> > I changed the scrape_interval today forgot to change it in grafana >> > variable. >> > Thank you. >> >> You are likely to have issues with a scrape interval that long. >> >> Due to staleness the maximum scrape interval is about 2 minutes, so >> you'd be best off reducing it from 5m. >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5d6c6648-daaa-4622-bc8c-06f6daff228a%40googlegroups.com.
