[
https://issues.apache.org/jira/browse/BEAM-5778?focusedWorklogId=163123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-163123
]
ASF GitHub Bot logged work on BEAM-5778:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Nov/18 17:00
Start Date: 06/Nov/18 17:00
Worklog Time Spent: 10m
Work Description: lgajowy commented on a change in pull request #6943:
[BEAM-5778] Add integrations of Metrics API to Big Query for SyntheticcSources
load tests in Python SDK
URL: https://github.com/apache/beam/pull/6943#discussion_r231201616
##########
File path:
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
##########
@@ -21,16 +21,121 @@
from __future__ import absolute_import
+import logging
import time
+from google.cloud import bigquery
+from google.cloud.bigquery.schema import SchemaField
+
import apache_beam as beam
from apache_beam.metrics import Metrics
+START_TIME_LABEL = 'runtime_start'
+END_TIME_LABEL = 'runtime_end'
+RUNTIME_LABEL = 'runtime'
+SUBMIT_TIMESTAMP_LABEL = 'submit_timestamp'
+LOAD_TEST_DATASET_NAME = 'python_load_tests'
+
+
+def _get_schema_field(schema_field):
+ return SchemaField(
+ name=schema_field['name'],
+ field_type=schema_field['type'],
+ mode=schema_field['mode'])
+
+
+class BigQueryMetricsCollector(object):
Review comment:
How about splitting this code even more i.e.:
- have separate class for BigQuery that totally abstracts from the term
"metric" (eg. `BigQueryClient` class)
- have a high level way to obtain the metrics in a form we need (eg.
`MetricsReader` class or such)
This way we could obtain metrics using the `MetricsReader` class and do
whatever we need with this data (eg. print to console). If we additionally want
to save them to BQ we could instantiate the `BigQueryPublisher` class and do it
easily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 163123)
Time Spent: 1h 40m (was: 1.5h)
> Add integrations of Metrics API to Big Query for SyntheticSources load tests
> in Python SDK
> ------------------------------------------------------------------------------------------
>
> Key: BEAM-5778
> URL: https://issues.apache.org/jira/browse/BEAM-5778
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Kasia Kucharczyk
> Assignee: Kasia Kucharczyk
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Right now Metrics API collects basic metrics of load tests of
> SyntheticSources (Python SDK). It should be collected in BigQuery for
> presenting it on performance dashboards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
