That looks fine to me. The "head block" contains all timeseries which were active in the last two hours; if you have some series churn (i.e. new series being created, old ones which stop) then the head block will grow as new series are added, but you'll only see the old ones drop out when it makes a new block. You can clearly see those dips in the green graph; I'd expect the same in the yellow one at 12:45 and 16:45 but they're just out of frame.
Since the two prometheus instances don't change head block at the same time, the number of timeseries in each are different. - green swapped at about 14:08, then starts to grow - yellow swapped at about 14:45, dropping to the level where green was at 14:08, and then starts to grow from there On Monday, 23 January 2023 at 14:41:24 UTC vikas....@gmail.com wrote: > We have 2 replicas of prometheus agent setup in kubernetes cluster. > > We noticed that both these prometheus agents are not scraping same number > of metrics. this we verified using prometheus_agent_active_series metrics. > > However when checked for prometheus agent targets its exactly same for > both agents. the difference in prometheus_agent_active_series is consistent > (attached is prometheus_agent_active_series graph) > > [image: Screenshot 2023-01-18 at 7.05.59 PM.png] > > We were expecting prometheus_agent_active_series should match for both > agents. not sure if the prometheus_agent_active_series is not reliable or > something else. > > please let me know what else can I check to drill this down? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/afe0298f-e1db-4288-a1df-4d59698d0f8an%40googlegroups.com.
