kkucharc commented on a change in pull request #11567:
URL: https://github.com/apache/beam/pull/11567#discussion_r422124124
##########
File path: sdks/python/apache_beam/testing/load_tests/load_test.py
##########
@@ -45,6 +47,19 @@ def _add_argparse_args(cls, parser):
'--metrics_table',
help='A BigQuery table where metrics should be '
'written.')
+ parser.add_argument(
+ '--influx_measurement',
+ help='An InfluxDB measurement where metrics should be published to. If
'
Review comment:
I am not sure if I correctly understand what measurement means. Is it
name for metric such as `runtime` or name of place where metric will be stored
as "table" or "column"?
##########
File path: sdks/python/apache_beam/testing/load_tests/load_test.py
##########
@@ -67,22 +82,30 @@ def _str_to_boolean(value):
class LoadTest(object):
+ """Base class for all integration and performance tests which export
+ metrics to external databases: BigQuery or/and InfluxDB.
+
+ Refer to :class:`~apache_beam.testing.load_tests.LoadTestOptions` for more
+ information on the required pipeline options.
+
+ If using InfluxDB with Basic HTTP authentication enabled, provide the
+ following environment options: `INFLUXDB_USER` and `INFLUXDB_USER_PASSWORD`.
Review comment:
Is it something we could enable to provide via PipelineOptions as well?
##########
File path: sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
##########
@@ -167,14 +175,15 @@ class MetricsReader(object):
A :class:`MetricsReader` retrieves metrics from pipeline result,
prepares it for publishers and setup publishers.
"""
- publishers = [] # type: List[ConsoleMetricsPublisher]
+ publishers = [] # type: List[Any]
def __init__(
self,
project_name=None,
bq_table=None,
bq_dataset=None,
publish_to_bq=False,
Review comment:
Do you think it would be good to have consistent parameter naming for
influx and bq? Or we plan to abandon bq in future?
##########
File path: sdks/python/apache_beam/testing/load_tests/load_test.py
##########
@@ -67,22 +82,30 @@ def _str_to_boolean(value):
class LoadTest(object):
+ """Base class for all integration and performance tests which export
+ metrics to external databases: BigQuery or/and InfluxDB.
+
+ Refer to :class:`~apache_beam.testing.load_tests.LoadTestOptions` for more
+ information on the required pipeline options.
+
+ If using InfluxDB with Basic HTTP authentication enabled, provide the
+ following environment options: `INFLUXDB_USER` and `INFLUXDB_USER_PASSWORD`.
+ """
def __init__(self):
# Be sure to set blocking to false for timeout_ms to work properly
self.pipeline = TestPipeline(is_integration_test=True, blocking=False)
assert not self.pipeline.blocking
- load_test_options = self.pipeline.get_pipeline_options().view_as(
- LoadTestOptions)
- self.timeout_ms = load_test_options.timeout_ms
- self.input_options = load_test_options.input_options
- self.metrics_namespace = load_test_options.metrics_table or 'default'
- publish_to_bq = load_test_options.publish_to_big_query
+ options = self.pipeline.get_pipeline_options().view_as(LoadTestOptions)
+ self.timeout_ms = options.timeout_ms
+ self.input_options = options.input_options
+ self.metrics_namespace = options.metrics_table or 'default'
+ publish_to_bq = options.publish_to_big_query
if publish_to_bq is None:
Review comment:
Maybe we should remove this `if` since we have now two targets where we
publish metrics?
##########
File path: sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
##########
@@ -404,6 +419,77 @@ def save(self, results):
return self._client.insert_rows(self._bq_table, results)
+class InfluxDBMetricsPublisherOptions(object):
+ def __init__(
+ self,
+ measurement, # type: str
+ db_name, # type: str
+ hostname='http://localhost:8086', # type: str
Review comment:
Why do we need this default value here? Isn't it provided from pipeline
options default value?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
