To make things more clear, this is not a problem for live-monitoring with short retention time. The problem that we are facing is a long-time trend analysis. For this, we export all Prometheus data in InfluxDB. For long retention times this become a real problem because the time series cardinality grows indefinitely leading to indefinite growth of the memory usage, since InfluxDB keeps in-memory index of all tags. In 2 weeks on 30GB RAM VM the memory usage grew from 47% to 75%. On Tuesday, September 12, 2017 at 12:36:43 PM UTC+2, Matthias Rampke wrote: > A rescheduled pod is a new pod, and there is no logical continuity with any > specific predecessor. Prometheus 1.x was designed with this in mind – one of > the main motivations for developing it were the difficulties we (SoundCloud) > had with per-instance time series in Graphite. However, it's not perfect – > frequently churning time series still come at a cost. Prometheus 2.0 > (currently in beta) will improve this significantly. > > > I would caution against micro-optimising for the frequency of time series > changes. If pods live on the order of hours, you are well within what > Prometheus 1.x supports, it just needs a little more resources to deal with > it. If this doesn't work for you, try if the Prometheus 2.x beta works for > you. > > > /MR > > > PS: cAdvisor/kubelet do sometimes create per-container metrics that vary > across restarts, or did in the past. You can try using relabelling to reduce > these, but you risk causing duplicate time series where the old and new > container overlap, and that's not good either. I tried this and rolled it > back for this reason. > > > > > > > On Tue, Sep 12, 2017 at 6:07 AM <sergei.st...@gmail.com> wrote: > Can you elaborate on this, please > > On Tuesday, September 12, 2017 at 12:24:28 AM UTC+2, Tim Hockin wrote: > > > When it is rescheduled, it very likely ends up on a different Node. > > > If you want to erase that info, you'll need to track ordinals yourself > > > (via templating or via an ID service) or use StatefulSet. > > > > > > On Mon, Sep 11, 2017 at 1:03 PM, <sergei.st...@gmail.com> wrote: > > > > Indeed, restarts do not create new series. However, we don't want a new > > > serie even when the pods get rescheduled. > > > > On Monday, September 11, 2017 at 9:41:01 PM UTC+2, Tim Hockin wrote: > > > >> Pod restarts should not create new series'. Only if they get > > >> rescheduled, as in a rolling update. In that case they ARE different. > > > >> > > > >> > > > >> > > > >> On Sep 6, 2017 2:06 AM, <sergei.st...@gmail.com> wrote: > > > >> Is there any way to create ordinal index of pods in a normal ReplicaSet > > >> similarly to StatefulSet? > > > >> > > > >> From functional point of view this doesn't make much sense. But could be > > >> really useful for our monitoring solution. We use Prometheus that > > >> scrapes metrics from kubernetes cluster and sends it to InfluxDB. Each > > >> newly created pod gets a unique name by adding a random suffix. The > > >> problem is that each time a new pod of the same replica is created, a > > >> new time series is created. This increases the metrics cardinality > > >> reducing performance and increasing memory footprint. It also results in > > >> creating each time a new line in a dashboard instead of continuing the > > >> same line. > > > >> > > > >> For example, suppose, we have a ReplicaSet with 2 replicas and each > > >> replica had 5 restarts. This will result in 10 separate time series and > > >> 10 lines in a monitoring dashboard, while with the ordinal index in > > >> place, there will be just 2. > > > >> > > > >> When dealing with very long time intervals, hundred/thousands time > > >> series will be created for the same instance instead of one. > > > >> > > > >> Any suggestions are greatly appreciated > > > >> > > > >> > > > >> > > > >> -- > > > >> > > > >> You received this message because you are subscribed to the Google > > >> Groups "Kubernetes user discussion and Q&A" group. > > > >> > > > >> To unsubscribe from this group and stop receiving emails from it, send > > >> an email to kubernetes-use...@googlegroups.com. > > > >> > > > >> To post to this group, send email to kubernet...@googlegroups.com. > > > >> > > > >> Visit this group at https://groups.google.com/group/kubernetes-users. > > > >> > > > >> For more options, visit https://groups.google.com/d/optout. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > "Kubernetes user discussion and Q&A" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to kubernetes-use...@googlegroups.com. > > > > To post to this group, send email to kubernet...@googlegroups.com. > > > > Visit this group at https://groups.google.com/group/kubernetes-users. > > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-use...@googlegroups.com. > > To post to this group, send email to kubernet...@googlegroups.com. > > Visit this group at https://groups.google.com/group/kubernetes-users. > > For more options, visit https://groups.google.com/d/optout.
-- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
