Hi Rob, I would also like metadata to become stateless, and view 6815 <https://github.com/prometheus/prometheus/pull/6815> only as a first step, and the start of an output format. Currently, there is a work in progress design doc, and another topic for an upcoming dev summit, for allowing use cases where metadata needs to be in the same request as the samples.
Generally, I (and some others I have talked to) don't want to send all the metadata with every sample as that is very repetitive, specifically for histograms and metrics with many series. Instead, I would like remote write requests to become transaction based, at which point all the metadata from that scrape/transaction can be added to the metadata field introduced to the proto in 6815 <https://github.com/prometheus/prometheus/pull/6815> and then each sample can be linked to a metadata entry without as much duplication. That is very broad strokes, and I am sure it will be refined or changed completely with more usage. That said, TYPE and UNIT are much smaller than metric name and help text, and I would support adding those to a linked metadata entry before remote write becomes transactional. Would that satisfy your use cases? Chris On Thu, Jul 16, 2020 at 1:43 PM Rob Skillington <[email protected]> wrote: > Typo: "community request" should be: "community contribution that > duplicates some of PR 6815" > > On Thu, Jul 16, 2020 at 3:27 PM Rob Skillington <[email protected]> > wrote: > >> Firstly: Thanks a lot for sharing the dev summit notes, they are greatly >> appreciated. Also thank you for a great PromCon! >> >> In regards to prometheus remote write metadata propagation consensus, is >> there any plans/projects/collaborations that can be done to perhaps plan >> work on a protocol that might help others in the ecosystem offer the same >> benefits to Prometheus ecosystem projects that operate on a per write >> request basis (i.e. stateless processing of a write request)? >> >> I understand https://github.com/prometheus/prometheus/pull/6815 unblocks >> feature development on top of Prometheus for users with specific >> architectures, however it is a non-starter for a lot of other projects, >> especially for third party exporters to systems that are unowned by end >> users (i.e. writing a StackDriver remote write endpoint that targeted >> StackDriver, the community is unable to change the implementation of >> StackDriver itself to cache/statefully make metrics metadata available at >> ingestion time to StackDriver). >> >> Obviously I have a vested interest since as a remote write target, M3 has >> several stateless components before TSDB ingestion and flowing the entire >> metadata to a distributed set of DB nodes that own a different set of the >> metrics space from each other node this has implications on M3 itself of >> course too (i.e. it is non-trivial to map metric name -> DB node without >> some messy stateful cache sitting somewhere in the architecture which adds >> operational burdens to end users). >> >> I suppose what I'm asking is, are maintainers open to a community request >> that duplicates some of >> https://github.com/prometheus/prometheus/pull/6815 but sends just metric >> TYPE and UNIT per datapoint (which would need to be captured by the WAL if >> feature is enabled) to a backend so it can statefully be processed >> correctly without needing a sync of a global set of metadata to a backend? >> >> And if not, what are the plans here and how can we collaborate to make >> this data useful to other consumers in the Prometheus ecosystem. >> >> Best intentions, >> Rob >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CABakzZbvZeyKLXfK08aiXgGcZso%3D8A0H1JBT9jwBzf6rCiUmVw%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-developers/CABakzZbvZeyKLXfK08aiXgGcZso%3D8A0H1JBT9jwBzf6rCiUmVw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CANVFovU9LKp6gvhu-hEEAsKxoY9jz0ONUdPES%2BOQ7hpwVnpY%2Bg%40mail.gmail.com.
