[
https://issues.apache.org/jira/browse/BEAM-11092?focusedWorklogId=513028&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-513028
]
ASF GitHub Bot logged work on BEAM-11092:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Nov/20 16:52
Start Date: 17/Nov/20 16:52
Worklog Time Spent: 10m
Work Description: pabloem commented on a change in pull request #13217:
URL: https://github.com/apache/beam/pull/13217#discussion_r525289585
##########
File path: sdks/python/apache_beam/internal/metrics/metric.py
##########
@@ -136,3 +165,64 @@ def log_metrics(self, reset_after_logging=False):
self._last_logging_millis = current_millis
finally:
self._lock.release()
+
+
+class ServiceCallMetric(object):
+ """Metric class which records Service API call metrics.
+
+ This class will capture a request count metric for the specified
+ request_count_urn and base_labels.
+
+ When call() is invoked the status must be provided, which will
+ be converted to a canonical GCP status code, if possible.
+
+ TODO(ajamato): Add Request latency metric.
+ """
+ def __init__(self, request_count_urn, base_labels=None):
+ # type: (str, Optional[Dict[str, str]]) -> None
Review comment:
fyi - from now on, you can use type hints directly, not as comments (no
need to change anything atm)
##########
File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py
##########
@@ -566,10 +568,30 @@ def _insert_all_rows(
skipInvalidRows=skip_invalid_rows,
# TODO(silviuc): Should have an option for ignoreUnknownValues?
rows=rows))
+
+ resource = '//bigquery.googleapis.com/projects/%s/datasets/%s/tables/%s' %
(
+ project_id, dataset_id, table_id)
+
+ labels = {
+ # TODO(ajamato): Add Ptransform label.
+ monitoring_infos.SERVICE_LABEL: 'BigQuery',
+ monitoring_infos.METHOD_LABEL: 'BigQueryBatchWrite',
Review comment:
The method here would be a BatchWrite? Or a Streaming Insert maybe?
##########
File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py
##########
@@ -566,10 +568,30 @@ def _insert_all_rows(
skipInvalidRows=skip_invalid_rows,
# TODO(silviuc): Should have an option for ignoreUnknownValues?
rows=rows))
+
+ resource = '//bigquery.googleapis.com/projects/%s/datasets/%s/tables/%s' %
(
+ project_id, dataset_id, table_id)
Review comment:
Maybe add the resource calculation to a helper method? So it can be
reused later? (I imagine you'll add these metrics for other methods?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 513028)
Time Spent: 2h 20m (was: 2h 10m)
> Add support for GCP IO metrics
> ------------------------------
>
> Key: BEAM-11092
> URL: https://issues.apache.org/jira/browse/BEAM-11092
> Project: Beam
> Issue Type: Test
> Components: beam-model
> Reporter: Alex Amato
> Assignee: Alex Amato
> Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This involves, collecting new metrics from the IO libraries. And adding
> "process wide" "harness metric" APIs. Metrics which are not bound to a bundle.
> [_https://s.apache.org/beam-gcp-debuggability_]
> In addition to adding a histogram style metrics.
> [https://s.apache.org/beam-histogram-metrics]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
