[
https://issues.apache.org/jira/browse/BEAM-4775?focusedWorklogId=201153&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201153
]
ASF GitHub Bot logged work on BEAM-4775:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Feb/19 08:56
Start Date: 20/Feb/19 08:56
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #7876: [BEAM-4775]
Clean up metric protos; support integer distributions, gauges
URL: https://github.com/apache/beam/pull/7876#discussion_r258384748
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -501,48 +501,9 @@ message MonitoringInfoTypeUrns {
message Metric {
// (Required) The data for this metric.
oneof data {
- CounterData counter_data = 1;
- DistributionData distribution_data = 2;
- ExtremaData extrema_data = 3;
- }
-}
-
-// Data associated with a Counter or Gauge metric.
-// This is designed to be compatible with metric collection
-// systems such as DropWizard.
-message CounterData {
- oneof value {
- int64 int64_value = 1;
- double double_value = 2;
- string string_value = 3;
- }
-}
-
-// Extrema messages are used for calculating
-// Top-N/Bottom-N metrics.
-message ExtremaData {
- oneof extrema {
- IntExtremaData int_extrema_data = 1;
- DoubleExtremaData double_extrema_data = 2;
- }
-}
-
-message IntExtremaData {
- repeated int64 int_values = 1;
-}
-
-message DoubleExtremaData {
- repeated double double_values = 2;
-}
-
-// Data associated with a distribution metric.
-// This is based off of the current DistributionData metric.
-// This is not a stackdriver or dropwizard compatible
-// style of distribution metric.
-message DistributionData {
- oneof distribution {
- IntDistributionData int_distribution_data = 1;
- DoubleDistributionData double_distribution_data = 2;
+ int64 counter = 1;
Review comment:
I don't remember discussing this, it must have been with someone else.
I agree that supporting double counters (and especially distributions) would
be useful. Also, we haven't really nailed down what the global semantics are
for gauge metrics.
My position was, and remains, that reifying the exact set of supported
metric types via a set of nested set of protos is not a stable, long-term
solution. It took a long time to come up with this list, and here we are 6
months later changing it. (And change will become harder the more we actually
use these protos everywhere.) Instead, data should be either (1) a bytes field
or (2) something like
message data {
repeated int64 int_data = 1;
repeated double double_data = 2;
repeated string string_data = 3;
...?
}
with the type specifying its encoding into this data attribute. We can then
support everything from simple counters to complex distributions without
changing the protos, only introducing new types, where the set of types are
defined by URNs, and runners simply pass through types they do not understand.
This is consistent with the rest of the way the FnAPI works as well (e.g.
coders, transforms, ...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 201153)
Time Spent: 15h 10m (was: 15h)
> JobService should support returning metrics
> -------------------------------------------
>
> Key: BEAM-4775
> URL: https://issues.apache.org/jira/browse/BEAM-4775
> Project: Beam
> Issue Type: Bug
> Components: beam-model
> Reporter: Eugene Kirpichov
> Assignee: Ryan Williams
> Priority: Major
> Labels: triaged
> Time Spent: 15h 10m
> Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_job_api.proto]
> currently doesn't appear to have a way for JobService to return metrics to a
> user, even though
> [https://github.com/apache/beam/blob/master/model/fn-execution/src/main/proto/beam_fn_api.proto]
> includes support for reporting SDK metrics to the runner harness.
>
> Metrics are apparently necessary to run any ValidatesRunner tests because
> PAssert needs to validate that the assertions succeeded. However, this
> statement should be double-checked: perhaps it's possible to somehow work
> with PAssert without metrics support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
