Are you building with -tags stringlabels? This tends to reduce memory used by Labels. Also I hope to reduce memory much more, though work stalled for a bit. (PR <https://github.com/prometheus/prometheus/pull/12304>)
Since Prometheus has to store all the output series in the head, I wouldn't take Cortex ruler impact as a guide to Prometheus impact. But if you measure that please do update us. I expect we would accept a PR adding an interface, unless it impacted performance noticeably. Can you share a pointer to your Cortex branch? Bryan (Prometheus maintainer, ex-Cortex maintainer) On Monday, 4 September 2023 at 16:14:40 UTC+1 Noam Dishon wrote: > Hello, > > I’m from Microsoft managed Prometheus team. We are running multiple cortex > ruler instances. > I’ve noticed that a large percentage of the memory of the ruler is > invested in keeping track of stale marks for series that no longer appear > in the result set. > > The complete set of labels of each iteration (of each rule, of each rule > group) are stored in memory, in seriesInPreviousEval > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L325C8-L325C8>, > > until the next iteration. They are then compared > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L755C13-L755C13> > > to the current iteration to see if any series is no longer returns from the > query. If so, that series is sent to the appender to be marked as stale. > > > I did a POC where the seriesInPreviousEval of each rule in stored in a > disk temp file. The results shows a reduction of about 75% in memory. > There is also a reduction of around 30% in CPU. No apparent negative affect > on evaluation duration. > > [image: Picture1.png] > [image: Picture2.png] > [image: Picture3.png] > [image: Picture4.png] > [image: Picture5.png] > > (These results are for cortex ruler, which does only rule processing, so > the effect is dramatic. But I’m positive that a standalone Prometheus > installation which handles a large number of rules will also benefit from > different handling of seriesInPreviousEval.) > > > > I would like to propose a couple of changes in this area: > > 2 1. Create an interface “PreviousTimeseriesStore” (or any better > name…) > > 2 2. Create a default implementation > MemoryPreviousTimeseriesStore. It would be similar to the current one. > However, I do think we can improve the memory implementation. It currently > is based on map[string]labels.Labels > <https://github.com/prometheus/prometheus/blob/95e705612c1d557f1681bd081a841b78f93ee158/rules/manager.go#L739> > > . The string part is made from the Bytes > <https://github.com/prometheus/prometheus/blob/b6f903b5f92b5458ad2244d9f442f7f859c01eb3/model/labels/labels.go#L72C33-L72C33> > > of the labels. > > I think we can use instead a map[string]bool (effectively a hash-set). > > Only if the labels are needed (which is just when a series doesn’t appear > in the current result set) then we need the labels set, and it can then be > reconstructed from the string. It requires FromBytes function in Labels > pkg, which isn’t complicated. > > C 3. Create another implementation which stores the series into the > disk. The same string of each series will be stored in a file as lines. > Switching to this implementation will be done via configuration. > > > 4. Custom implementation of the interface could be passed in > ManagerOptions > > > > Thanks, > Noam > > > > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6666c947-352f-4f0e-9f19-ff9b94c149ben%40googlegroups.com.
