subject:"\[jira\] \[Commented\] \(BEAM\-1866\) FnAPI support for Metrics"

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

2018-01-05 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313491#comment-16313491
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

robertwb closed pull request #4344: [BEAM-1866] Plumb user metrics through Fn 
API.
URL: https://github.com/apache/beam/pull/4344
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto 
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index ca23c619f33..36ed4242d6c 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -280,11 +280,44 @@ message Metrics {
 
   // User defined metrics
   message User {
-// TODO: Define it.
+
+// A key for identifying a metric at the most granular level.
+message MetricKey {
+  // The step, if any, this metric is associated with.
+  string step = 1;
+
+  // (Required): The namespace of this metric.
+  string namespace = 2;
+
+  // (Required): The name of this metric.
+  string name = 3;
+}
+
+// Data associated with a counter metric.
+message CounterData {
+  int64 value = 1;
+}
+
+// Data associated with a distribution metric.
+message DistributionData {
+  int64 count = 1;
+  int64 sum = 2;
+  int64 min = 3;
+  int64 max = 4;
+}
+
+// (Required) The identifier for this metric.
+MetricKey key = 1;
+
+// (Required) The data for this metric.
+oneof data {
+  CounterData counter_data = 1001;
+  DistributionData distribution_data = 1002;
+}
   }
 
   map ptransforms = 1;
-  map user = 2;
+  repeated User user = 2;
 }
 
 message ProcessBundleProgressResponse {
diff --git a/sdks/python/apache_beam/metrics/cells.py 
b/sdks/python/apache_beam/metrics/cells.py
index ba840f752b1..2b5a4e4094d 100644
--- a/sdks/python/apache_beam/metrics/cells.py
+++ b/sdks/python/apache_beam/metrics/cells.py
@@ -28,6 +28,7 @@
 
 from apache_beam.metrics.metricbase import Counter
 from apache_beam.metrics.metricbase import Distribution
+from apache_beam.portability.api import beam_fn_api_pb2
 
 __all__ = ['DistributionResult']
 
@@ -286,10 +287,18 @@ def combine(self, other):
 new_min,
 new_max)
 
-  @classmethod
-  def singleton(cls, value):
+  @staticmethod
+  def singleton(value):
 return DistributionData(value, 1, value, value)
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.DistributionData(
+count=self.count, sum=self.sum, min=self.min, max=self.max)
+
+  @staticmethod
+  def from_runner_api(proto):
+return DistributionData(proto.sum, proto.count, proto.min, proto.max)
+
 
 class MetricAggregator(object):
   """For internal use only; no backwards-compatibility guarantees.
diff --git a/sdks/python/apache_beam/metrics/execution.py 
b/sdks/python/apache_beam/metrics/execution.py
index 1704b98d46a..9e547a983de 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -34,6 +34,8 @@
 
 from apache_beam.metrics.cells import CounterCell
 from apache_beam.metrics.cells import DistributionCell
+from apache_beam.metrics.metricbase import MetricName
+from apache_beam.portability.api import beam_fn_api_pb2
 
 
 class MetricKey(object):
@@ -63,6 +65,14 @@ def __str__(self):
   def __hash__(self):
 return hash((self.step, self.metric))
 
+  def to_runner_api(self):
+return beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step, namespace=self.metric.namespace, name=self.metric.name)
+
+  @staticmethod
+  def from_runner_api(proto):
+return MetricKey(proto.step, MetricName(proto.namespace, proto.name))
+
 
 class MetricResult(object):
   """Keeps track of the status of a metric within a single bundle.
@@ -192,6 +202,20 @@ def get_cumulative(self):
 """
 return self._get_updates()
 
+  def to_runner_api(self):
+return (
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+counter_data=beam_fn_api_pb2.Metrics.User.CounterData(
+value=v.get_cumulative()))
+ for k, v in self.counters.items()] +
+[beam_fn_api_pb2.Metrics.User(
+key=beam_fn_api_pb2.Metrics.User.MetricKey(
+step=self.step_name, namespace=k.namespace, name=k.name),
+distribution_data=v.get_cumulative().to_runner_api())
+ for k, v in self.distributions.items()])
+
 
 class ScopedMetricsContainer(object):
 
diff --git

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

2018-01-04 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312041#comment-16312041
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

robertwb opened a new pull request #4344: [BEAM-1866] Plumb user metrics 
through Fn API.
URL: https://github.com/apache/beam/pull/4344
 
 
   The SDK worker is now periodically querried for progress
   and user metrics.
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>  Labels: portability
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954384#comment-15954384
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2412


> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api
>Reporter: Daniel Halperin
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

2017-04-03 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954063#comment-15954063
 ] 

ASF GitHub Bot commented on BEAM-1866:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2412

[BEAM-1866] DataflowRunner: disable PAssert use of metrics when FnAPI is 
enabled

FnApi does not yet support metrics

R: @tgroh 

CC: @pabloem Pablo, I think the work you're doing to improve `PAssert` 
might be useful here.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam fnapi-passert

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2412


commit 0a3a8b918ce66f11c610ce1948ce290e5f699520
Author: Dan Halperin 
Date:   2017-04-03T19:46:33Z

DataflowRunner: disable PAssert use of metrics when FnAPI is enabled

FnApi does not yet support metrics




> FnAPI support for Metrics
> -
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-fn-api
>Reporter: Daniel Halperin
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

[jira] [Commented] (BEAM-1866) FnAPI support for Metrics

4 matches

Site Navigation

Mail list logo

Footer information