Re: About the Gauge metric API

Interesting point. Some runners might implement metrics via a control plane, in which case per-key stuff is problematic for large numbers of keys. However other runners may decide to implement metrics inside the graph itself (e.g. by generating a CombinePerKey), in which case per-key aggregation

Re: About the Gauge metric API

Great summary overall. A few small things to add, along with a note that more examples of the intended is for each metric/aggregation may be helpful. It is worth looking at what existing metric systems provide. Specifically, two things to consider: 1. Is scoping implict / automatic or explicit.

Re: About the Gauge metric API

On Fri, Apr 6, 2018 at 12:23 PM Ben Chambers wrote: > Generally strong +1 to everything Bill said. I would suggest though that > the per-worker segmentation might be specified using some more general > tagging/labeling API. For instance, all of the following seem like >

Re: About the Gauge metric API

Generally strong +1 to everything Bill said. I would suggest though that the per-worker segmentation might be specified using some more general tagging/labeling API. For instance, all of the following seem like reasonable uses to support: 1. Gauge that is tagged with worker to get per-worker

Re: About the Gauge metric API

In terms of natural language, I don't think "gauge" makes sense strings. A gauge measures a quantity. A string is not a quantity. So I like a separate API, like Robert says. Backends can go ahead and implement leaf String and Gauge collectors with the same data structure if they like. In

Re: About the Gauge metric API

Thanks for unraveling those themes, Pablo! 1. Seems reasonable in light of behaviors metrics backends support. 2. Those same backends support histogramming of data, so having integer types is very useful. 3. I believe that is the case, for the reasons I mentioned earlier, Gauges should only

Re: About the Gauge metric API

Nobody wants to get rid of Gauges. I see that we have three separate themes being discussed here, and I think it's useful to point them out and address them independently: 1. Whether Gauges should change to hold string values. 2. If Gauges are to support string values, whether Gauges should also

Re: About the Gauge metric API

Gauges are incredibly useful for exposing the current state of the system. For instance, number of elements in a queue, current memory usage, number of RPCs in flight, etc. As mentioned above, these concepts exist in numerous systems for monitoring distributed environments, including Stackdriver

Re: About the Gauge metric API

A gauge API is only useful if there's a correlation to a distributed worker, because "clobber" is not a useful aggregation method. There's no useful correlation of that signal to anything actionable. Ben's already noted this point that metrics backends do this, but it seems that if gauge is to be

Re: About the Gauge metric API

Just naively - the use cases that Gauge addresses seem relevant, and the information seems feasible to gather and present. The bit that doesn't seem to make sense is aggregating gauges by clobbering each other. So I think that's just +1 Ben? On Fri, Apr 6, 2018 at 10:26 AM Raghu Angadi

Re: About the Gauge metric API

I am not opposed to removing other data types, though they are extra convenience for user. In Scott's example above, if the metric is a counter, what are the guarantees provided? E.g. would it match the global count using GBK? If yes, then gauges (especially per-key gauges) can be very useful too

Re: About the Gauge metric API

> > I would be in favor of replacing the existing Gauge.set(long) API with the > String version and removing the old one. This would be a breaking change. > However this is a relatively new API and is still marked @Experimental. > Keeping the old API would retain the potential confusion. It's

Re: About the Gauge metric API

Hi Ben : D Sure, that's reasonable. And perhaps I started the discussion in the wrong direction. I'm not questioning the utility of Gauge metrics. What I'm saying is that Beam only supports integers,, but Gauges are aggregated by dropping old values depending on their update times; so it might

Re: About the Gauge metric API

Some metrics backend label the value, for instance with the worker that sent it. Then the aggregation is latest per label. This makes it useful for holding values such as "memory usage" that need to hold current value. On Fri, Apr 6, 2018, 9:00 AM Scott Wegner wrote: > +1 on

Re: About the Gauge metric API

+1 on the proposal to support a "String" gauge. To expand a bit, the current API doesn't make it clear that the gauge value is based on local state. If a runner chooses to parallelize a DoFn across many workers, each worker will have its own local Gauge metric and its updates will overwrite other