AnandInguva commented on code in PR #25242:
URL: https://github.com/apache/beam/pull/25242#discussion_r1097898583
##########
sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py:
##########
@@ -32,6 +35,27 @@
_OUTPUT_GCS_BUCKET_ROOT = 'gs://temp-storage-for-end-to-end-tests/tft/'
+def _publish_metrics(pipeline, metric_value, metrics_table):
+ influx_options = InfluxDBMetricsPublisherOptions(
+ metrics_table,
+ pipeline.get_option('influx_db_name'),
+ pipeline.get_option('influx_hostname'),
+ os.getenv('INFLUXDB_USER'),
+ os.getenv('INFLUXDB_USER_PASSWORD'),
+ )
+ metric_reader = MetricsReader(
+ project_name=pipeline.get_option('project'),
+ bq_table=metrics_table,
+ bq_dataset=pipeline.get_option('metrics_dataset'),
+ publish_to_bq=True,
+ influxdb_options=influx_options,
+ )
+ metric_reader.publish_values([(
+ 'runtime',
+ metric_value,
Review Comment:
1. Changed it.
2. Yes, that is right, once the changes are in, I will create a grafana
dashboard.
3. I am not sure if that would be useful. MetricsReader is all we need and
to publish it to influx DB we would need InfluxDB related information, which I
passed it through pipeline options. Other benchmark tests, they have an
abstract class `LoadTest`, which takes care of these things. Other benchmark
test like fastavro_it_test do it like the way I did it.
4. Added a param to the function to accept any metric name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
