On Sun, 21 Jun 2020 at 01:44, Bashar Al Rawi <[email protected]> wrote:
> Hi all, > > I would like to propose a new format for storing bucket counts that can > provide substantial improvements for sparse/bi-modal metrics along with > changes to histogram_quantile that allow the new format. Both the change > and format aren't complex and can be done with backward compatibility. The > main breaking change would be adding a new reserved label. > > I put together my thoughts in this Google doc > <https://docs.google.com/document/d/1aYWDLs15j-rlH2nLyoNimj7SP2RGGw_fe2ST9OqZeDI/edit?usp=sharing> > that > is open for comments and a rough implementation in this commit > <https://github.com/basharal/prometheus/commit/fba202e04f32724a2dcd03d78da35a0a6eda81d3> > . > Thanks for your doc. The first challenge here is that this involves a format change. The Prometheus text format is basically fixed at this point, OpenMetrics will be replacing it but sticks with the same format for histograms as OpenMetrics aims to leverage the existing installed base. The duplication in the current format isn't really a problem in practice, as compression handles this. This proposal also loses the benefits of the current format, namely that you can no longer drop overly-granular buckets nor can you calculate things like Apdex both of which are important properties. So this would break important use cases, and cause busy work across the ecosystem. The really big issue here is that understanding is that even with something like this you still need ~300 buckets to not have to worry about bucket choices given the typical range of event sizes. This is well above the ~10 recommended currently when considering other labels, so something would need to be done to make the TSDB handling of histograms ~30x more efficient before this would make sense. The way to approach this instead is to leave the format as-is, and look at how to optimise this inside the TSDB (and later PromQL) so that more buckets are handled transparently and efficiently for histogram_quantile - but otherwise look the same to PromQL as today. In short changing the text format - even with it being a major breaking change that'd break key use cases - would be the easy bit, and doesn't actually help. The hard bits are all in the TSDB, and you should try to solve those first as without TSDB improvements there's no point. Brian > > Please, take a look and let me know what your thoughts are. > > Bashar > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/b3cd52af-3e60-4385-8a0d-b914559ab0b0n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/b3cd52af-3e60-4385-8a0d-b914559ab0b0n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLo459pk_ex1cBjDAe_3AjVNUQ6wrSCEKfq%3DhGQ%2B-fnnfQ%40mail.gmail.com.
