Interesting point. Some runners might implement metrics via a control
plane, in which case per-key stuff is problematic for large numbers of
keys. However other runners may decide to implement metrics inside the
graph itself (e.g. by generating a CombinePerKey), in which case per-key
aggregation is
Great summary overall. A few small things to add, along with a note that
more examples of the intended is for each metric/aggregation may be helpful.
It is worth looking at what existing metric systems provide. Specifically,
two things to consider:
1. Is scoping implict / automatic or explicit. I
On Fri, Apr 6, 2018 at 12:23 PM Ben Chambers wrote:
> Generally strong +1 to everything Bill said. I would suggest though that
> the per-worker segmentation might be specified using some more general
> tagging/labeling API. For instance, all of the following seem like
> reasonable uses to support
Generally strong +1 to everything Bill said. I would suggest though that
the per-worker segmentation might be specified using some more general
tagging/labeling API. For instance, all of the following seem like
reasonable uses to support:
1. Gauge that is tagged with worker to get per-worker segme
In terms of natural language, I don't think "gauge" makes sense strings. A
gauge measures a quantity. A string is not a quantity. So I like a separate
API, like Robert says. Backends can go ahead and implement leaf String and
Gauge collectors with the same data structure if they like.
In implement
Thanks for unraveling those themes, Pablo!
1. Seems reasonable in light of behaviors metrics backends support.
2. Those same backends support histogramming of data, so having integer
types is very useful.
3. I believe that is the case, for the reasons I mentioned earlier, Gauges
should only clobbe
Nobody wants to get rid of Gauges. I see that we have three separate themes
being discussed here, and I think it's useful to point them out and address
them independently:
1. Whether Gauges should change to hold string values.
2. If Gauges are to support string values, whether Gauges should also
c
Gauges are incredibly useful for exposing the current state of the system.
For instance, number of elements in a queue, current memory usage, number
of RPCs in flight, etc. As mentioned above, these concepts exist in
numerous systems for monitoring distributed environments, including
Stackdriver Mo
A gauge API is only useful if there's a correlation to a distributed
worker, because "clobber" is not a useful aggregation method. There's no
useful correlation of that signal to anything actionable. Ben's already
noted this point that metrics backends do this, but it seems that if gauge
is to be a
Just naively - the use cases that Gauge addresses seem relevant, and the
information seems feasible to gather and present. The bit that doesn't seem
to make sense is aggregating gauges by clobbering each other. So I think
that's just +1 Ben?
On Fri, Apr 6, 2018 at 10:26 AM Raghu Angadi wrote:
>
I am not opposed to removing other data types, though they are extra
convenience for user.
In Scott's example above, if the metric is a counter, what are the
guarantees provided? E.g. would it match the global count using GBK? If
yes, then gauges (especially per-key gauges) can be very useful too
A String API makes it clear(er) that the values will not be aggregated in
any way across workers. I don't think retaining both APIs (except for
possibly some short migration period) worthwhile. On another note, I still
find the distributed gague API to be a bit odd in general.
On Fri, Apr 6, 2018
>
> I would be in favor of replacing the existing Gauge.set(long) API with the
> String version and removing the old one. This would be a breaking change.
> However this is a relatively new API and is still marked @Experimental.
> Keeping the old API would retain the potential confusion. It's bette
Hi Ben : D
Sure, that's reasonable. And perhaps I started the discussion in the wrong
direction. I'm not questioning the utility of Gauge metrics.
What I'm saying is that Beam only supports integers,, but Gauges are
aggregated by dropping old values depending on their update times; so it
might be
See for instance how gauge metrics are handled in Prometheus, Datadog and
Stackdriver monitoring. Gauges are perfect for use in distributed systems,
they just need to be properly labeled. Perhaps we should apply a default
tag or allow users to specify one.
On Fri, Apr 6, 2018, 9:14 AM Ben Chambers
Some metrics backend label the value, for instance with the worker that
sent it. Then the aggregation is latest per label. This makes it useful for
holding values such as "memory usage" that need to hold current value.
On Fri, Apr 6, 2018, 9:00 AM Scott Wegner wrote:
> +1 on the proposal to supp
+1 on the proposal to support a "String" gauge.
To expand a bit, the current API doesn't make it clear that the gauge value
is based on local state. If a runner chooses to parallelize a DoFn across
many workers, each worker will have its own local Gauge metric and its
updates will overwrite other
Hello all,
As I was working on adding support for Gauges in Dataflow, some noted that
Gauge is a fairly unusual kind of metric for a distributed environment,
since many workers will report different values and stomp on each other's
all the time.
We also looked at Flink and Dropwizard Gauge metrics
18 matches
Mail list logo
