On Tue, Jun 22, 2021 at 12:09 PM Richard Hartmann < richih.mailingl...@gmail.com> wrote:
> This has come up in the context of OM, OTel, and TAG Observability. My > own thinking largely mirrors beorn's & grobie's: In a perfect world > the orchestration layer has all the information and interfaces > required and billing knows about the required datapaths, NB: > Monitoring usually has higher speed and lower reliability requirements > than billing. Still, for doability, lock-in, convenience, and velocity > reasons, it's enticing to bypass the ideal solution and do something > that works-ish now. If someone incurs ~100% overhead for monitoring > lightweight functions but gets their job done, they are are still > getting their job done and can optimize later if they so choose. > > Pushing might appear hamfisted here, and arguably is, but it's largely > under the control of the dev; as such, they can do it with less > coordination. This might get us near to using the Prometheus Agent as > a Collector to reduce latency and blast radius. Far from ideal, but... > > An in-between would be what grobie said: To speak in Prometheus terms, > the orchestrator is node_exporter, the serverless functions write out > something which the textfile collector can ingest. > There is not much overlap between the node_exporter and the functionality needed here. It would need something which can read common log streams from major cloud providers / serverless runtimes, aggregate the logs, and then expose them. Only the last part is somewhat available in the node_exporter and the rest doesn't really make sense there. Google's mtail would be a bit closer conceptually, but as we have full control over the clients and wire format there is no need for a full-fledged log parsing engine, and the cloud provider log reading part is still missing. OpenMetrics deliberately supports push, but this approach creates > issues with `up` and staleness handling. OTel is currently facing > similar issues, maybe there's room for cooperation. Also see > > https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems > and > https://docs.google.com/document/d/1hn-u6WKLHxIsqYT1_u6eh94lyQeXrFaAouMshJcQFXs/edit#heading=h.e4p9f543e7i2 > > > I strongly believe that we should be particular about the wire format; > in a future in which orchestrators have a collector component, it > would be nice to be able to simply expose the metrics for pulling or > use PRW code and wire format. > > > Best, > Richard > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/CAD77%2BgSiKWVrnoGydB2hBVkeX87NejCht93JPVvaY%2BQ-Y%3DGvoQ%40mail.gmail.com > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAChBsdAC6LSeR%2BQDpmpae-Ty27KeQq-nSo6mdGdhxV3bQiJ-yg%40mail.gmail.com.
