[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445554688



##
File path: 
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBPublisher.java
##
@@ -19,14 +19,16 @@
 
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static java.util.Objects.requireNonNull;
-import static 
org.apache.beam.repackaged.core.org.apache.commons.lang3.StringUtils.isNoneBlank;
 
 import java.io.IOException;
 import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.repackaged.core.org.apache.commons.lang3.StringUtils;

Review comment:
   ah..I would swear I changed it...thanks!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445451720



##
File path: 
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
##
@@ -142,21 +148,27 @@ void runAll(String[] args) throws IOException {
 saveSummary(null, configurations, actual, baseline, start, options);
   }
 
-  if (options.getExportSummaryToBigQuery()) {
-ImmutableMap schema =
-ImmutableMap.builder()
-.put("timestamp", "timestamp")
-.put("runtimeSec", "float")
-.put("eventsPerSec", "float")
-.put("numResults", "integer")
-.build();
+  final ImmutableMap schema =
+  ImmutableMap.builder()
+  .put("timestamp", "timestamp")
+  .put("runtimeSec", "float")

Review comment:
   Sure, but I think we need to change wording to `runtimeMs` as well, WDYT?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445395098



##
File path: 
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
##
@@ -142,21 +148,27 @@ void runAll(String[] args) throws IOException {
 saveSummary(null, configurations, actual, baseline, start, options);
   }
 
-  if (options.getExportSummaryToBigQuery()) {
-ImmutableMap schema =
-ImmutableMap.builder()
-.put("timestamp", "timestamp")
-.put("runtimeSec", "float")
-.put("eventsPerSec", "float")
-.put("numResults", "integer")
-.build();
+  final ImmutableMap schema =
+  ImmutableMap.builder()
+  .put("timestamp", "timestamp")
+  .put("runtimeSec", "float")
+  .put("eventsPerSec", "float")

Review comment:
   Sure, I'll remove it from implementation but not here (we want to 
preserve compatibility with BQ), I'll change it in influx publisher, thanks for 
the info!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445395098



##
File path: 
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
##
@@ -142,21 +148,27 @@ void runAll(String[] args) throws IOException {
 saveSummary(null, configurations, actual, baseline, start, options);
   }
 
-  if (options.getExportSummaryToBigQuery()) {
-ImmutableMap schema =
-ImmutableMap.builder()
-.put("timestamp", "timestamp")
-.put("runtimeSec", "float")
-.put("eventsPerSec", "float")
-.put("numResults", "integer")
-.build();
+  final ImmutableMap schema =
+  ImmutableMap.builder()
+  .put("timestamp", "timestamp")
+  .put("runtimeSec", "float")
+  .put("eventsPerSec", "float")

Review comment:
   Sure, I'll remove it from implementation but not here, I'll change it in 
influx publisher, thanks for the tip!

##
File path: 
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
##
@@ -142,21 +148,27 @@ void runAll(String[] args) throws IOException {
 saveSummary(null, configurations, actual, baseline, start, options);
   }
 
-  if (options.getExportSummaryToBigQuery()) {
-ImmutableMap schema =
-ImmutableMap.builder()
-.put("timestamp", "timestamp")
-.put("runtimeSec", "float")
-.put("eventsPerSec", "float")
-.put("numResults", "integer")
-.build();
+  final ImmutableMap schema =
+  ImmutableMap.builder()
+  .put("timestamp", "timestamp")
+  .put("runtimeSec", "float")
+  .put("eventsPerSec", "float")

Review comment:
   Sure, I'll remove it from implementation but not here, I'll change it in 
influx publisher, thanks for the info!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445336567



##
File path: 
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
##
@@ -142,21 +148,27 @@ void runAll(String[] args) throws IOException {
 saveSummary(null, configurations, actual, baseline, start, options);
   }
 
-  if (options.getExportSummaryToBigQuery()) {
-ImmutableMap schema =
-ImmutableMap.builder()
-.put("timestamp", "timestamp")
-.put("runtimeSec", "float")
-.put("eventsPerSec", "float")
-.put("numResults", "integer")
-.build();
+  final ImmutableMap schema =
+  ImmutableMap.builder()
+  .put("timestamp", "timestamp")
+  .put("runtimeSec", "float")
+  .put("eventsPerSec", "float")
+  .put("numResults", "integer")
+  .build();
 
+  if (options.getExportSummaryToBigQuery()) {
 savePerfsToBigQuery(
 BigQueryResultsPublisher.create(options.getBigQueryDataset(), 
schema),
 options,
 actual,
 start);
   }
+
+  if (options.getExportSummaryToInfluxDB()) {
+final long timestamp = start.getMillis() / 1000; // seconds

Review comment:
   The default precision is nanoseconds. In case of nexmark results we 
changed it and use seconds instead
   ```
   return new HttpPost(
   settings.host + "/write?db=" + settings.database + "&" + 
retentionPolicy + "=s");
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-25 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r445331560



##
File path: 
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBPublisher.java
##
@@ -66,30 +125,43 @@ private static void publish(
   builder.setDefaultCredentialsProvider(provider);
 }
 
-final HttpPost postRequest = new HttpPost(settings.host + "/write?db=" + 
settings.database);
+return builder;
+  }
 
-final StringBuilder metricBuilder = new StringBuilder();
-results.stream()

Review comment:
   Hm...sure we can.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam] pawelpasterz commented on a change in pull request #11956: [BEAM-8133] Publishing results of Nexmark tests to InfluxDB

2020-06-17 Thread GitBox 


pawelpasterz commented on a change in pull request #11956:
URL: https://github.com/apache/beam/pull/11956#discussion_r441368908



##
File path: 
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBPublisher.java
##
@@ -40,34 +43,72 @@
 
   private InfluxDBPublisher() {}
 
+  public static void publishNexmarkResults(
+  final Collection> results, final InfluxDBSettings 
settings) {
+publishWithCheck(settings, () -> publishNexmark(results, settings));
+  }
+
   public static void publishWithSettings(
   final Collection results, final InfluxDBSettings 
settings) {
+publishWithCheck(settings, () -> publishCommon(results, settings));
+  }
+
+  private static void publishWithCheck(
+  final InfluxDBSettings settings, final PublishFunction publishFunction) {
 requireNonNull(settings, "InfluxDB settings must not be null");
 if (isNoneBlank(settings.measurement, settings.database)) {
   try {
-publish(results, settings);
-  } catch (final Exception exception) {
+publishFunction.publish();
+  } catch (Exception exception) {
 LOG.warn("Unable to publish metrics due to error: {}", 
exception.getMessage(), exception);
   }
 } else {
   LOG.warn("Missing property -- measurement/database. Metrics won't be 
published.");
 }
   }
 
-  private static void publish(
-  final Collection results, final InfluxDBSettings 
settings) throws Exception {
+  private static void publishNexmark(
+  final Collection> results, final InfluxDBSettings 
settings)
+  throws Exception {
 
-final HttpClientBuilder builder = HttpClientBuilder.create();
+final HttpClientBuilder builder = provideHttpBuilder(settings);
+final HttpPost postRequest = providePOSTRequest(settings);
+final StringBuilder metricBuilder = new StringBuilder();
+results.forEach(
+map ->
+metricBuilder
+.append(map.get("measurement"))

Review comment:
   Yeah, makes sense, thanks!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org