Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3672

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4393

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1454

[Apache Jenkins Server]

See 


Changes:

[klk] Add GAUGE_DATA case to metricUpdatesFromProto

[klk] Add Gauge metric tests to ensure value persists

--
[...truncated 95.22 KB...]
'apache-beam-testing:bqjob_r7cf3303101183bc_016213b68398_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 06:19:12,990 712024d5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 06:19:32,710 712024d5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 06:19:34,872 712024d5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.13s,  CPU:0.24s,  MaxMemory:25216kb 
STDOUT: Upload complete.
Waiting on bqjob_r36d288e4f1ccce6b_016213b6d906_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r36d288e4f1ccce6b_016213b6d906_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r36d288e4f1ccce6b_016213b6d906_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 06:19:34,873 712024d5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 06:19:52,250 712024d5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 06:19:54,427 712024d5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.17s,  CPU:0.27s,  MaxMemory:25520kb 
STDOUT: Upload complete.
Waiting on bqjob_r72604354f1b74a18_016213b7255f_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r72604354f1b74a18_016213b7255f_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r72604354f1b74a18_016213b7255f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 06:19:54,428 712024d5 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 06:20:24,018 712024d5 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 06:20:26,196 712024d5 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.16s,  CPU:0.38s,  MaxMemory:25524kb 
STDOUT: Upload complete.
Waiting on bqjob_r70b41e12c51adc4e_016213b7a1d9_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r70b41e12c51adc4e_016213b7a1d9_1 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5213

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1010

[Apache Jenkins Server]

See 


Changes:

[klk] Add GAUGE_DATA case to metricUpdatesFromProto

[klk] Add Gauge metric tests to ensure value persists

--
[...truncated 1.11 KB...]
 > git rev-list --no-walk be520691cd80b8b49b98474bab1c05dbd7baac68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8116099075571940453.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8726971827003947272.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6685272221618610076.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6883046135346276988.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5016404303043643058.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2680457371452593531.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #238

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5131

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5130

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5212

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4392

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3671

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4391

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3670

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5211

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1453

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for ARRAY expression

[kedin] [SQL] Support ARRAY in projections

[kedin] [SQL] Implement array elements access expression

[kedin] [SQL] Add support for ELEMENT(collection) function

[kedin] [SQL] Add support for CARDINALITY(collection)

[apilloud] Dataflow runner must supply attempted metrics

[apilloud] Throw UnsupportedOperationException instead of returning null

--
[...truncated 89.45 KB...]
'apache-beam-testing:bqjob_r2d3c5d05a4a96c1a_0162126bc11f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 00:17:56,491 47af0305 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 00:18:20,486 47af0305 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 00:18:22,807 47af0305 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.31s,  CPU:0.26s,  MaxMemory:25272kb 
STDOUT: Upload complete.
Waiting on bqjob_r2833767c90de908d_0162126c27e2_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r2833767c90de908d_0162126c27e2_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2833767c90de908d_0162126c27e2_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 00:18:22,807 47af0305 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 00:18:40,789 47af0305 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 00:18:43,115 47af0305 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.31s,  CPU:0.26s,  MaxMemory:25276kb 
STDOUT: Upload complete.
Waiting on bqjob_rb49256d9cd9972b_0162126c7764_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_rb49256d9cd9972b_0162126c7764_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_rb49256d9cd9972b_0162126c7764_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-11 00:18:43,116 47af0305 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-11 00:19:07,074 47af0305 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-11 00:19:09,447 47af0305 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #8

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79269
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 11/Mar/18 00:11
Start Date: 11/Mar/18 00:11
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4848: [BEAM-1866] 
fixups to gauge metrics translation
URL: https://github.com/apache/beam/pull/4848
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
index 744bd53f5eb..596fe231fcd 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
@@ -23,7 +23,6 @@
 import com.google.common.cache.LoadingCache;
 import java.util.ArrayList;
 import java.util.Collection;
-import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi;
@@ -42,7 +41,7 @@ public static MetricUpdates metricUpdatesFromProto(
   String ptransformName, Collection 
userMetricUpdates) {
 List counterUpdates = new ArrayList<>();
 List distributionUpdates = 
new ArrayList<>();
-List gaugeUpdates = 
Collections.emptyList();
+List gaugeUpdates = new 
ArrayList<>();
 
 for (BeamFnApi.Metrics.User userMetricUpdate : userMetricUpdates) {
   MetricKey metricKey =
@@ -63,6 +62,13 @@ public static MetricUpdates metricUpdatesFromProto(
   userMetricUpdate.getDistributionData().getMin(),
   userMetricUpdate.getDistributionData().getMax(;
   break;
+case GAUGE_DATA:
+  gaugeUpdates.add(
+  MetricUpdates.MetricUpdate.create(
+  metricKey,
+  GaugeData.create(
+  userMetricUpdate.getGaugeData().getValue(;
+  break;
 case DATA_NOT_SET:
   continue;
   }
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
index 4e5edc36f03..936067afe3d 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
@@ -38,6 +38,7 @@
   // Transform names are arbitrary user-meaningful steps in processing
   private static final String TRANSFORM1 = "transform1";
   private static final String TRANSFORM2 = "transform2";
+  private static final String TRANSFORM3 = "transform3";
 
   // Namespaces correspond to different contexts for a metric
   private static final String NAMESPACE1 = "fakeNamespace1";
@@ -48,6 +49,8 @@
   private static final String COUNTER_NAME2 = "dropped";
   private static final String DISTRIBUTION_NAME1 = "someMillis";
   private static final String DISTRIBUTION_NAME2 = "otherMillis";
+  private static final String GAUGE_NAME1 = "load";
+  private static final String GAUGE_NAME2 = "memory";
 
   private static final BeamFnApi.Metrics.User.MetricName COUNTER_METRIC1 =
   BeamFnApi.Metrics.User.MetricName.newBuilder()
@@ -73,6 +76,18 @@
   .setName(DISTRIBUTION_NAME2)
   .build();
 
+  private static final BeamFnApi.Metrics.User.MetricName GAUGE_METRIC1 =
+  BeamFnApi.Metrics.User.MetricName.newBuilder()
+  .setNamespace(NAMESPACE1)
+  .setName(GAUGE_NAME1)
+  .build();
+
+  private static final BeamFnApi.Metrics.User.MetricName GAUGE_METRIC2 =
+  BeamFnApi.Metrics.User.MetricName.newBuilder()
+  .setNamespace(NAMESPACE2)
+  .setName(GAUGE_NAME2)
+  .build();
+
   private static final BeamFnApi.Metrics.User DISTRIBUTION1 =
   BeamFnApi.Metrics.User.newBuilder()
   .setMetricName(DISTRIBUTION_METRIC1)
@@ -107,6 +122,18 @@
   
.setCounterData(BeamFnApi.Metrics.User.CounterData.newBuilder().setValue(0L))
   .build();
 
+  private static final BeamFnApi.Metrics.User GAUGE1 =
+  BeamFnApi.Metrics.User.newBuilder()
+  .setMetricName(GAUGE_METRIC2)
+  
.setCounterData(BeamFnApi.Metrics.User.CounterData.newBuilder().setValue(56L))
+  .build();
+
+  

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79268
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 11/Mar/18 00:11
Start Date: 11/Mar/18 00:11
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4831: [BEAM-1866] Add 
GAUGE_DATA case to metricUpdatesFromProto
URL: https://github.com/apache/beam/pull/4831
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
index 744bd53f5eb..014163bc91e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
@@ -62,6 +62,12 @@ public static MetricUpdates metricUpdatesFromProto(
   userMetricUpdate.getDistributionData().getCount(),
   userMetricUpdate.getDistributionData().getMin(),
   userMetricUpdate.getDistributionData().getMax(;
+case GAUGE_DATA:
+  gaugeUpdates.add(
+  MetricUpdates.MetricUpdate.create(
+  metricKey,
+  GaugeData.create(
+  userMetricUpdate.getGaugeData().getValue(;
   break;
 case DATA_NOT_SET:
   continue;
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
index 4e5edc36f03..df1a654785b 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
@@ -38,6 +38,7 @@
   // Transform names are arbitrary user-meaningful steps in processing
   private static final String TRANSFORM1 = "transform1";
   private static final String TRANSFORM2 = "transform2";
+  private static final String TRANSFORM2 = "transform3";
 
   // Namespaces correspond to different contexts for a metric
   private static final String NAMESPACE1 = "fakeNamespace1";
@@ -48,6 +49,8 @@
   private static final String COUNTER_NAME2 = "dropped";
   private static final String DISTRIBUTION_NAME1 = "someMillis";
   private static final String DISTRIBUTION_NAME2 = "otherMillis";
+  private static final String GAUGE_NAME1 = "load";
+  private static final String GAUGE_NAME2 = "memory";
 
   private static final BeamFnApi.Metrics.User.MetricName COUNTER_METRIC1 =
   BeamFnApi.Metrics.User.MetricName.newBuilder()
@@ -72,6 +75,18 @@
   .setNamespace(NAMESPACE2)
   .setName(DISTRIBUTION_NAME2)
   .build();
+  
+  private static final BeamFnApi.Metrics.User.MetricName GAUGE_METRIC1 =
+  BeamFnApi.Metrics.User.MetricName.newBuilder()
+  .setNamespace(NAMESPACE1)
+  .setName(GAUGE_NAME1)
+  .build();
+
+  private static final BeamFnApi.Metrics.User.MetricName GAUGE_METRIC2 =
+  BeamFnApi.Metrics.User.MetricName.newBuilder()
+  .setNamespace(NAMESPACE2)
+  .setName(GAUGE_NAME2)
+  .build();
 
   private static final BeamFnApi.Metrics.User DISTRIBUTION1 =
   BeamFnApi.Metrics.User.newBuilder()
@@ -106,6 +121,18 @@
   .setMetricName(COUNTER_METRIC2)
   
.setCounterData(BeamFnApi.Metrics.User.CounterData.newBuilder().setValue(0L))
   .build();
+  
+  private static final BeamFnApi.Metrics.User GAUGE1 =
+  BeamFnApi.Metrics.User.newBuilder()
+  .setMetricName(GAUGE_METRIC2)
+  
.setCounterData(BeamFnApi.Metrics.User.CounterData.newBuilder().setValue(56L))
+  .build();
+
+  private static final BeamFnApi.Metrics.User GAUGE2 =
+  BeamFnApi.Metrics.User.newBuilder()
+  .setMetricName(GAUGE_METRIC2)
+  
.setCounterData(BeamFnApi.Metrics.User.CounterData.newBuilder().setValue(3L))
+  .build();
 
   @Parameterized.Parameters
   public static Iterable testInstances() {
@@ -125,6 +152,13 @@
   ImmutableMap.builder()
   .put(TRANSFORM1, ImmutableList.of(DISTRIBUTION1, COUNTER1))
   .put(TRANSFORM2, ImmutableList.of(COUNTER2))
+  .put(TRANSFORM3, ImmutableList.of(GAUGE1))
+  .build()
+})
+.add(
+new Object[] {
+  

[beam] branch master updated (be52069 -> 33c368c)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from be52069  Merge pull request #4802: [BEAM-3785][SQL] Add support for 
arrays
 add 36a9fa1  Add GAUGE_DATA case to metricUpdatesFromProto
 add 90148b8  Add Gauge metric tests to ensure value persists
 new 33c368c  Merge pull request #4848: [BEAM-1866] fixups to gauge metrics 
translation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/core/metrics/MetricsTranslation.java   | 10 +--
 .../core/metrics/MetricsTranslationTest.java   | 34 ++
 2 files changed, 42 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4848: [BEAM-1866] fixups to gauge metrics translation

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 33c368c77a8e82194cbee095b27641cf673751cd
Merge: be52069 90148b8
Author: Kenn Knowles 
AuthorDate: Sat Mar 10 16:11:28 2018 -0800

Merge pull request #4848: [BEAM-1866] fixups to gauge metrics translation

 .../runners/core/metrics/MetricsTranslation.java   | 10 +--
 .../core/metrics/MetricsTranslationTest.java   | 34 ++
 2 files changed, 42 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #1009

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for ARRAY expression

[kedin] [SQL] Support ARRAY in projections

[kedin] [SQL] Implement array elements access expression

[kedin] [SQL] Add support for ELEMENT(collection) function

[kedin] [SQL] Add support for CARDINALITY(collection)

[apilloud] Dataflow runner must supply attempted metrics

[apilloud] Throw UnsupportedOperationException instead of returning null

--
[...truncated 1.19 KB...]
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins595486603370532309.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8028003288282968617.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5744814319490527083.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins383085685808972670.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/ad/dc/fcced9ec3f2561c0cbe8eb6527eef7cf4f4919a2b3a07891a36e846635af/setuptools-38.5.2-py2.py3-none-any.whl#md5=abd3307cdce6fb543b5a4d0e3e98bdb6
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5885901975152091258.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3491171533443935415.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #237

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for ARRAY expression

[kedin] [SQL] Support ARRAY in projections

[kedin] [SQL] Implement array elements access expression

[kedin] [SQL] Add support for ELEMENT(collection) function

[kedin] [SQL] Add support for CARDINALITY(collection)

[apilloud] Dataflow runner must supply attempted metrics

[apilloud] Throw UnsupportedOperationException instead of returning null

--
[...truncated 678.99 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #239

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79266
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 10/Mar/18 23:30
Start Date: 10/Mar/18 23:30
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4848: [BEAM-1866] 
fixups to gauge metrics translation
URL: https://github.com/apache/beam/pull/4848#issuecomment-372075867
 
 
   Confirmed green after fixing the typos. Squashing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79266)
Time Spent: 3h  (was: 2h 50m)

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Priority: Major
>  Labels: portability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5129

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3789) [SQL] Support Nested Rows

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3789?focusedWorklogId=79262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79262
 ]

ASF GitHub Bot logged work on BEAM-3789:


Author: ASF GitHub Bot
Created on: 10/Mar/18 22:47
Start Date: 10/Mar/18 22:47
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #4822: [BEAM-3789][SQL] 
Support nested Rows
URL: https://github.com/apache/beam/pull/4822#issuecomment-372073418
 
 
   Rebased, added tests for array fields


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79262)
Time Spent: 1h  (was: 50m)

> [SQL] Support Nested Rows
> -
>
> Key: BEAM-3789
> URL: https://issues.apache.org/jira/browse/BEAM-3789
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add support for SqlTypeName.ROW



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3789) [SQL] Support Nested Rows

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3789?focusedWorklogId=79261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79261
 ]

ASF GitHub Bot logged work on BEAM-3789:


Author: ASF GitHub Bot
Created on: 10/Mar/18 22:47
Start Date: 10/Mar/18 22:47
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #4822: [BEAM-3789][SQL] 
Support nested Rows
URL: https://github.com/apache/beam/pull/4822#issuecomment-372073418
 
 
   rebased added tests for array fields


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79261)
Time Spent: 50m  (was: 40m)

> [SQL] Support Nested Rows
> -
>
> Key: BEAM-3789
> URL: https://issues.apache.org/jira/browse/BEAM-3789
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add support for SqlTypeName.ROW



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4390

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3669

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5210

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4389

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5128

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5209

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3789) [SQL] Support Nested Rows

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3789?focusedWorklogId=79253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79253
 ]

ASF GitHub Bot logged work on BEAM-3789:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:29
Start Date: 10/Mar/18 20:29
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4822: [BEAM-3789][SQL] 
Support nested Rows
URL: https://github.com/apache/beam/pull/4822#issuecomment-372063632
 
 
   https://reviewable.io/lgtm.png; height="20" width="61"/>
   
   ---
   
   Reviewed 8 of 8 files at r1.
   Review status: all files reviewed at latest revision, 3 unresolved 
discussions.
   
   ---
   
   
*[sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java,
 line 44 at 
r1](https://reviewable.io:443/reviews/apache/beam/4822#-L7G_-tB2t85K0bzwecj:-L7G_-tB2t85K0bzweck:b4c4jgi)
 ([raw 
file](https://github.com/apache/beam/blob/26105626e2bc2fa88f95d71be5583ee70b66e275/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java#L44)):*
   > ```Java
   > .BeamSqlPlusExpression;
   > import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.comparison
   > .BeamSqlEqualsExpression;
   > ```
   
   This is a bit weird
   
   ---
   
   
*[sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/package-info.java,
 line 23 at 
r1](https://reviewable.io:443/reviews/apache/beam/4822#-L7G_Ckna86tlo3_jfCh:-L7G_Ckna86tlo3_jfCi:b-jj7gg9)
 ([raw 
file](https://github.com/apache/beam/blob/26105626e2bc2fa88f95d71be5583ee70b66e275/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/row/package-info.java#L23)):*
   > ```Java
   >  */
   > @DefaultAnnotation(NonNull.class)
   > package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.row;
   > ```
   
   I think we might be getting a bit extreme about nesting Java packages. I 
prefer to keep them as un-nested as possible so that package-private access is 
useful and it is easy to browse the whole system at a glance. Most tools are 
best at one directory at a time, up to a few dozen files.
   
   ---
   
   
*[sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java,
 line 200 at 
r1](https://reviewable.io:443/reviews/apache/beam/4822#-L7GaLhzvH_ioLa0XIUN:-L7GaLhzvH_ioLa0XIUO:b-4ek08r)
 ([raw 
file](https://github.com/apache/beam/blob/26105626e2bc2fa88f95d71be5583ee70b66e275/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java#L200)):*
   > ```Java
   > .apply(
   > BeamSql.query(
   > "SELECT `PCOLLECTION`.`f_nestedRow`.`f_nestedString` 
FROM PCOLLECTION"))
   > ```
   
   Now that there's also arrays, it would be nice to build some complex nesting 
and make sure it all works. This PR is fine now, but you do have to rebase to 
resolve the conflicts anyhow.
   
   ---
   
   
   *Comments from 
[Reviewable](https://reviewable.io:443/reviews/apache/beam/4822#-:-L7GadAerQFXHsVyFnWI:bnfp4nl)*
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79253)
Time Spent: 0.5h  (was: 20m)

> [SQL] Support Nested Rows
> -
>
> Key: BEAM-3789
> URL: https://issues.apache.org/jira/browse/BEAM-3789
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add support for SqlTypeName.ROW



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3828) Investigate performance of SQL expression evaluation

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3828:
-

 Summary: Investigate performance of SQL expression evaluation
 Key: BEAM-3828
 URL: https://issues.apache.org/jira/browse/BEAM-3828
 Project: Beam
  Issue Type: Wish
  Components: dsl-sql
Reporter: Kenneth Knowles


If I am reading correctly, in a SELECT expression basically a syntax tree 
traversal style interpretation is applied. We should see if this shows up in 
profiles and might be profitable to optimize.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3785) [SQL] Add support for arrays

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3785?focusedWorklogId=79252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79252
 ]

ASF GitHub Bot logged work on BEAM-3785:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:21
Start Date: 10/Mar/18 20:21
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4802: [BEAM-3785][SQL] 
Add support for arrays
URL: https://github.com/apache/beam/pull/4802
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index c638eb1fd46..3eed316d7b2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -310,6 +310,16 @@ public Builder addValues(Object ... values) {
   return addValues(Arrays.asList(values));
 }
 
+public Builder addArray(List values) {
+  this.values.add(values);
+  return this;
+}
+
+public Builder addArray(Object ... values) {
+  addArray(Arrays.asList(values));
+  return this;
+}
+
 public Row build() {
   checkNotNull(type);
 
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/RowSqlType.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/RowSqlType.java
index 1208c42f6ac..71ff22897a1 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/RowSqlType.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/RowSqlType.java
@@ -102,6 +102,10 @@ public Builder withTimestampField(String fieldName) {
   return withField(fieldName, SqlTypeCoders.TIMESTAMP);
 }
 
+public Builder withArrayField(String fieldName, SqlTypeCoder elementCoder) 
{
+  return withField(fieldName, SqlTypeCoders.arrayOf(elementCoder));
+}
+
 private Builder() {
   this.fields = ImmutableList.builder();
 }
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTypeCoder.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTypeCoder.java
index 8c311f54828..eefc8701960 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTypeCoder.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTypeCoder.java
@@ -21,6 +21,7 @@
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.util.Objects;
 import org.apache.beam.sdk.coders.BigDecimalCoder;
 import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
 import org.apache.beam.sdk.coders.BigEndianLongCoder;
@@ -28,6 +29,7 @@
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.ListCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 
 /**
@@ -55,7 +57,6 @@ public void verifyDeterministic() throws 
NonDeterministicException {
   @Override
   public boolean equals(Object other) {
 return other != null && this.getClass().equals(other.getClass());
-
   }
 
   @Override
@@ -63,7 +64,12 @@ public int hashCode() {
 return this.getClass().hashCode();
   }
 
+  public static boolean isArray(SqlTypeCoder sqlTypeCoder) {
+return sqlTypeCoder instanceof SqlArrayCoder;
+  }
+
   static class SqlTinyIntCoder extends SqlTypeCoder {
+
 @Override
 protected Coder delegateCoder() {
   return ByteCoder.of();
@@ -153,4 +159,43 @@ protected Coder delegateCoder() {
   return RowHelper.DateCoder.of();
 }
   }
+
+  /**
+   * Represents SQL ARRAY type.
+   *
+   * Delegates to {#code elementCoder} to encode elements.
+   */
+  public static class SqlArrayCoder extends SqlTypeCoder {
+
+private SqlTypeCoder elementCoder;
+
+private SqlArrayCoder(SqlTypeCoder elementCoder) {
+  this.elementCoder = elementCoder;
+}
+
+public static SqlArrayCoder of(SqlTypeCoder elementCoder) {
+  return new SqlArrayCoder(elementCoder);
+}
+
+@Override
+protected Coder delegateCoder() {
+  return ListCoder.of(elementCoder);
+}
+
+public SqlTypeCoder getElementCoder() {
+  return elementCoder;
+}
+
+@Override
+public boolean equals(Object other) {
+  return other != null
+  && this.getClass().equals(other.getClass())
+  && this.elementCoder.equals(((SqlArrayCoder) other).elementCoder);
+}
+
+@Override
+public int hashCode() {
+  return 

[beam] branch master updated (20b13e9 -> be52069)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 20b13e9  Merge pull request #4841: BEAM-3803: Dataflow runner 
implements metrics contract
 add 594d5e1  [SQL] Add support for ARRAY expression
 add 925ca66  [SQL] Support ARRAY in projections
 add 21a6c78  [SQL] Implement array elements access expression
 add 581c5d1  [SQL] Add support for ELEMENT(collection) function
 add 5e032cb  [SQL] Add support for CARDINALITY(collection)
 new be52069  Merge pull request #4802: [BEAM-3785][SQL] Add support for 
arrays

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/values/Row.java  |  10 +
 .../apache/beam/sdk/extensions/sql/RowSqlType.java |   4 +
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  |  47 -
 .../beam/sdk/extensions/sql/SqlTypeCoders.java |   5 +
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  97 ++---
 .../interpreter/operator/BeamSqlPrimitive.java |   2 +
 .../BeamSqlArrayExpression.java}   |  36 ++--
 .../BeamSqlArrayItemExpression.java}   |  30 ++-
 .../interpreter/operator/array}/package-info.java  |   4 +-
 .../BeamSqlCardinalityExpression.java} |  31 ++-
 .../BeamSqlSingleElementExpression.java}   |  41 ++--
 .../operator/collection}/package-info.java |   4 +-
 .../extensions/sql/impl/utils/CalciteUtils.java|  68 +-
 .../beam/sdk/extensions/sql/BeamSqlArrayTest.java  | 228 +
 .../operator/array/BeamSqlArrayExpressionTest.java |  79 +++
 .../array/BeamSqlArrayItemExpressionTest.java  |  96 +
 .../BeamSqlCardinalityExpressionTest.java  |  91 
 .../BeamSqlSingleElementExpressionTest.java|  93 +
 18 files changed, 857 insertions(+), 109 deletions(-)
 copy 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/{math/BeamSqlRandExpression.java
 => array/BeamSqlArrayExpression.java} (68%)
 copy 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/{math/BeamSqlRandExpression.java
 => array/BeamSqlArrayItemExpression.java} (69%)
 copy {runners/core-java/src/main/java/org/apache/beam/runners/core => 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array}/package-info.java
 (88%)
 copy 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/{math/BeamSqlRandExpression.java
 => collection/BeamSqlCardinalityExpression.java} (69%)
 copy 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/{math/BeamSqlRandExpression.java
 => collection/BeamSqlSingleElementExpression.java} (57%)
 copy {runners/core-java/src/main/java/org/apache/beam/runners/core => 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection}/package-info.java
 (87%)
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlArrayTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayExpressionTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/array/BeamSqlArrayItemExpressionTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlCardinalityExpressionTest.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/collection/BeamSqlSingleElementExpressionTest.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4802: [BEAM-3785][SQL] Add support for arrays

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit be520691cd80b8b49b98474bab1c05dbd7baac68
Merge: 20b13e9 5e032cb
Author: Kenn Knowles 
AuthorDate: Sat Mar 10 12:21:31 2018 -0800

Merge pull request #4802: [BEAM-3785][SQL] Add support for arrays

 .../main/java/org/apache/beam/sdk/values/Row.java  |  10 +
 .../apache/beam/sdk/extensions/sql/RowSqlType.java |   4 +
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  |  47 -
 .../beam/sdk/extensions/sql/SqlTypeCoders.java |   5 +
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  97 ++---
 .../interpreter/operator/BeamSqlPrimitive.java |   2 +
 .../operator/array/BeamSqlArrayExpression.java |  57 ++
 .../operator/array/BeamSqlArrayItemExpression.java |  51 +
 .../interpreter/operator/array/package-info.java   |  26 +++
 .../collection/BeamSqlCardinalityExpression.java   |  52 +
 .../collection/BeamSqlSingleElementExpression.java |  64 ++
 .../operator/collection/package-info.java  |  26 +++
 .../extensions/sql/impl/utils/CalciteUtils.java|  68 +-
 .../beam/sdk/extensions/sql/BeamSqlArrayTest.java  | 228 +
 .../operator/array/BeamSqlArrayExpressionTest.java |  79 +++
 .../array/BeamSqlArrayItemExpressionTest.java  |  96 +
 .../BeamSqlCardinalityExpressionTest.java  |  91 
 .../BeamSqlSingleElementExpressionTest.java|  93 +
 18 files changed, 1058 insertions(+), 38 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-3785) [SQL] Add support for arrays

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3785?focusedWorklogId=79250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79250
 ]

ASF GitHub Bot logged work on BEAM-3785:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:18
Start Date: 10/Mar/18 20:18
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4802: [BEAM-3785][SQL] 
Add support for arrays
URL: https://github.com/apache/beam/pull/4802#issuecomment-372062689
 
 
   https://reviewable.io/lgtm.png; height="20" width="61"/> good to have this 
support in place. I expect that we will want to look at the performance of 
basically running a syntax-tree interpreter on every expression. Not new in 
this PR, but seeing the functionality added made me think of it. That's a slow 
approach in a tight loop.
   
   ---
   
   Reviewed 3 of 11 files at r1, 17 of 17 files at r2.
   Review status: all files reviewed at latest revision, all discussions 
resolved.
   
   ---
   
   
   
   *Comments from 
[Reviewable](https://reviewable.io:443/reviews/apache/beam/4802#-:-L7GYokZVVYlXVewqo3g:b-oakol6)*
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79250)
Time Spent: 1h  (was: 50m)

> [SQL] Add support for arrays
> 
>
> Key: BEAM-3785
> URL: https://issues.apache.org/jira/browse/BEAM-3785
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Support fields of Array type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3803) Dataflow runner should handle metrics per the spec

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3803?focusedWorklogId=79249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79249
 ]

ASF GitHub Bot logged work on BEAM-3803:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:11
Start Date: 10/Mar/18 20:11
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #4841: BEAM-3803: 
Dataflow runner implements metrics contract
URL: https://github.com/apache/beam/pull/4841
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
index 0983674b983..376befa77f4 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
@@ -153,9 +153,10 @@ public void addMetricResult(
 metricKey.metricName(),
 metricKey.stepName(),
 isStreamingJob ? null : value, // Committed
-isStreamingJob ? value : null)); // Attempted
+value)); // Attempted
 /* In Dataflow streaming jobs, only ATTEMPTED metrics are available.
- * In Dataflow batch jobs, only COMMITTED metrics are available.
+ * In Dataflow batch jobs, only COMMITTED metrics are available, but
+ * we must provide ATTEMPTED, so we use COMMITTED as a good 
approximation.
  * Reporting the appropriate metric depending on whether it's a 
batch/streaming job.
  */
   } else if (committed.getScalar() != null && attempted.getScalar() != 
null) {
@@ -166,9 +167,10 @@ public void addMetricResult(
 metricKey.metricName(),
 metricKey.stepName(),
 isStreamingJob ? null : value, // Committed
-isStreamingJob ? value : null)); // Attempted
+value)); // Attempted
 /* In Dataflow streaming jobs, only ATTEMPTED metrics are available.
- * In Dataflow batch jobs, only COMMITTED metrics are available.
+ * In Dataflow batch jobs, only COMMITTED metrics are available, but
+ * we must provide ATTEMPTED, so we use COMMITTED as a good 
approximation.
  * Reporting the appropriate metric depending on whether it's a 
batch/streaming job.
  */
   } else {
@@ -350,10 +352,18 @@ public static MetricQueryResults create(
 public abstract MetricName name();
 public abstract String step();
 @Nullable
-public abstract T committed();
-@Nullable
+protected abstract T committedInternal();
 public abstract T attempted();
 
+public T committed() {
+  T committed = committedInternal();
+  if (committed == null) {
+throw new UnsupportedOperationException("This runner does not 
currently support committed"
++ " metrics results. Please use 'attempted' instead.");
+  }
+  return committed;
+}
+
 public static  MetricResult create(MetricName name, String scope,
 T committed, T attempted) {
   return new AutoValue_DataflowMetrics_DataflowMetricResult<>(
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
index baf02114179..7aa291065aa 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
@@ -22,8 +22,10 @@
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.empty;
 import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.fail;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
@@ -208,7 +210,7 @@ public void testSingleCounterUpdates() throws IOException {
 DataflowMetrics dataflowMetrics = new DataflowMetrics(job, dataflowClient);
 MetricQueryResults result = dataflowMetrics.queryMetrics(null);
 assertThat(result.counters(), containsInAnyOrder(
-attemptedMetricsResult("counterNamespace", "counterName", 
"myStepName", (Long) null)));
+  

[beam] branch master updated (fc772c0 -> 20b13e9)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fc772c0  Merge pull request #4846 from charlesccychen/revert-pr-4666
 add 42a60a8  Dataflow runner must supply attempted metrics
 add d5d2177  Throw UnsupportedOperationException instead of returning null
 new 20b13e9  Merge pull request #4841: BEAM-3803: Dataflow runner 
implements metrics contract

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/dataflow/DataflowMetrics.java | 22 +
 .../beam/runners/dataflow/DataflowMetricsTest.java | 37 ++
 2 files changed, 40 insertions(+), 19 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4841: BEAM-3803: Dataflow runner implements metrics contract

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 20b13e9fdfd11d83321a4f75e0634c157718a791
Merge: fc772c0 d5d2177
Author: Kenn Knowles 
AuthorDate: Sat Mar 10 12:11:21 2018 -0800

Merge pull request #4841: BEAM-3803: Dataflow runner implements metrics 
contract

 .../beam/runners/dataflow/DataflowMetrics.java | 22 +
 .../beam/runners/dataflow/DataflowMetricsTest.java | 37 ++
 2 files changed, 40 insertions(+), 19 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79247
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:06
Start Date: 10/Mar/18 20:06
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #4848: 
[BEAM-1866] fixups to in gauge metrics translation
URL: https://github.com/apache/beam/pull/4848
 
 
   Fixes things on a LGTM PR adding Gauge translation case to metrics 
translation.
   
   Closes #4831


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79247)
Time Spent: 2h 40m  (was: 2.5h)

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79248=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79248
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 10/Mar/18 20:06
Start Date: 10/Mar/18 20:06
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4831: [BEAM-1866] Add 
GAUGE_DATA case to metricUpdatesFromProto
URL: https://github.com/apache/beam/pull/4831#issuecomment-372061719
 
 
   LGTM modulo minor typos that I've fixed in a derived PR. I'll go ahead and 
merge with fixups after getting green.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79248)
Time Spent: 2h 50m  (was: 2h 40m)

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1866) Fn API support for Metrics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1866?focusedWorklogId=79246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79246
 ]

ASF GitHub Bot logged work on BEAM-1866:


Author: ASF GitHub Bot
Created on: 10/Mar/18 19:27
Start Date: 10/Mar/18 19:27
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#4831: [BEAM-1866] Add GAUGE_DATA case to metricUpdatesFromProto
URL: https://github.com/apache/beam/pull/4831#discussion_r173632240
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
 ##
 @@ -38,6 +38,7 @@
   // Transform names are arbitrary user-meaningful steps in processing
   private static final String TRANSFORM1 = "transform1";
   private static final String TRANSFORM2 = "transform2";
+  private static final String TRANSFORM2 = "transform3";
 
 Review comment:
   Typo - `TRANSFORM3`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79246)
Time Spent: 2.5h  (was: 2h 20m)

> Fn API support for Metrics
> --
>
> Key: BEAM-1866
> URL: https://issues.apache.org/jira/browse/BEAM-1866
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Priority: Major
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> As part of the Fn API work, we need to define a Metrics interface between the 
> Runner and the SDK. Right now, Metrics are simply lost.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2083) Develop a Go SDK for Beam

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-2083:

Description: 
Allow users of the Go programming language (https://golang.org/) to write Beam 
pipelines in this language. The effort is focusing on full-fledged SDK that 
leverages the Beam Fn API to bootstrap a native Go experience.

Initial design:

https://s.apache.org/beam-go-sdk-design-rfc

Development in the master branch. Work in progress. YMMV.

  was:
Allow users of the Go programming language (https://golang.org/) to write Beam 
pipelines in this language. The effort is focusing on full-fledged SDK that 
leverages the Beam Fn API to bootstrap a native Go experience.

Initial design:

https://s.apache.org/beam-go-sdk-design-rfc

Development in the go-sdk branch. Work in progress. YMMV.


> Develop a Go SDK for Beam
> -
>
> Key: BEAM-2083
> URL: https://issues.apache.org/jira/browse/BEAM-2083
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Henning Rohde
>Priority: Major
>
> Allow users of the Go programming language (https://golang.org/) to write 
> Beam pipelines in this language. The effort is focusing on full-fledged SDK 
> that leverages the Beam Fn API to bootstrap a native Go experience.
> Initial design:
> https://s.apache.org/beam-go-sdk-design-rfc
> Development in the master branch. Work in progress. YMMV.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not

[Justin Tumale (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394296#comment-16394296
 ] 

Justin Tumale edited comment on BEAM-2817 at 3/10/18 6:42 PM:
--

This has been completed and merged. I think it will be a part of the 2.4.0 
release. PR #4606


was (Author: justintumale):
This has been completed and merged. I think it will be a part of the 2.4.0 
release.

> Bigquery queries should allow options to run in batch mode or not
> -
>
> Key: BEAM-2817
> URL: https://issues.apache.org/jira/browse/BEAM-2817
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.0.0
>Reporter: Lara Schmidt
>Assignee: Justin Tumale
>Priority: Major
>  Labels: newbie, starter
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When bigquery read does a query it sets the mode to batch. A batch query can 
> be very slow to schedule as it batches it with other queries. However it 
> doesn't use batch quota which is better for some cases. However, in some 
> cases a fast query is better (especially in timed tests). It would be a good 
> idea to have a configuration to the bigquery source to set this per-read.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not

[Justin Tumale (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394296#comment-16394296
 ] 

Justin Tumale commented on BEAM-2817:
-

This has been completed and merged. I think it will be a part of the 2.4.0 
release.

> Bigquery queries should allow options to run in batch mode or not
> -
>
> Key: BEAM-2817
> URL: https://issues.apache.org/jira/browse/BEAM-2817
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.0.0
>Reporter: Lara Schmidt
>Assignee: Justin Tumale
>Priority: Major
>  Labels: newbie, starter
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When bigquery read does a query it sets the mode to batch. A batch query can 
> be very slow to schedule as it batches it with other queries. However it 
> doesn't use batch quota which is better for some cases. However, in some 
> cases a fast query is better (especially in timed tests). It would be a good 
> idea to have a configuration to the bigquery source to set this per-read.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2817) Bigquery queries should allow options to run in batch mode or not

[Justin Tumale (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Tumale resolved BEAM-2817.
-
   Resolution: Done
Fix Version/s: 2.4.0

> Bigquery queries should allow options to run in batch mode or not
> -
>
> Key: BEAM-2817
> URL: https://issues.apache.org/jira/browse/BEAM-2817
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.0.0
>Reporter: Lara Schmidt
>Assignee: Justin Tumale
>Priority: Major
>  Labels: newbie, starter
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> When bigquery read does a query it sets the mode to batch. A batch query can 
> be very slow to schedule as it batches it with other queries. However it 
> doesn't use batch quota which is better for some cases. However, in some 
> cases a fast query is better (especially in timed tests). It would be a good 
> idea to have a configuration to the bigquery source to set this per-read.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4388

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1452

[Apache Jenkins Server]

See 


--
[...truncated 78.81 KB...]
'apache-beam-testing:bqjob_r1f1586a06cb2bd0_016211234d0d_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 18:19:10,910 51710360 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 18:19:33,310 51710360 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 18:19:35,742 51710360 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.42s,  CPU:0.32s,  MaxMemory:25472kb 
STDOUT: Upload complete.
Waiting on bqjob_r10955f886f454989_01621123adb6_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r10955f886f454989_01621123adb6_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r10955f886f454989_01621123adb6_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 18:19:35,743 51710360 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 18:19:54,513 51710360 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 18:19:56,837 51710360 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.31s,  CPU:0.27s,  MaxMemory:25420kb 
STDOUT: Upload complete.
Waiting on bqjob_r67e350624c303565_01621124002e_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r67e350624c303565_01621124002e_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r67e350624c303565_01621124002e_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 18:19:56,838 51710360 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 18:20:16,778 51710360 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 18:20:19,349 51710360 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.55s,  CPU:0.39s,  MaxMemory:25212kb 
STDOUT: Upload complete.
Waiting on bqjob_r3c254e9793b84ce5_0162112457c4_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r3c254e9793b84ce5_0162112457c4_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job

[jira] [Assigned] (BEAM-3545) Fn API metrics in Go SDK harness

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-3545:
---

Assignee: Robert Burke  (was: Henning Rohde)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3827) Add Go SDK integration tests

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-3827:
---

 Summary: Add Go SDK integration tests
 Key: BEAM-3827
 URL: https://issues.apache.org/jira/browse/BEAM-3827
 Project: Beam
  Issue Type: Task
  Components: sdk-go
Reporter: Henning Rohde
Assignee: Henning Rohde


We should add post-commit testing similar to ValidateRunner tests to ensure 
that the model is implemented correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3826) Complete Go SDK documentation

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-3826:
---

 Summary: Complete Go SDK documentation
 Key: BEAM-3826
 URL: https://issues.apache.org/jira/browse/BEAM-3826
 Project: Beam
  Issue Type: Task
  Components: sdk-go
Reporter: Henning Rohde
Assignee: Henning Rohde


We need to add a Go QuickStart and development guides on the website. The godoc 
should also be self-contained and published similarly to java and python api 
docs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3667

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1008

[Apache Jenkins Server]

See 


--
[...truncated 1.11 KB...]
 > git rev-list --no-walk fc772c08f62960b6b944b07bc38df20a1054d3f9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8403823704245201421.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4611930862882306864.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6794746034777378368.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins719824846597706355.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins775704316415026386.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7084876382201294764.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #7

[Apache Jenkins Server]

See 


--
[...truncated 9.12 KB...]
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #238

[Apache Jenkins Server]

See 


--
[...truncated 11.74 KB...]
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/usr/local/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.5.0.dev0)
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.5.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.5.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting proto-google-cloud-pubsub-v1==0.15.4 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.5.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/usr/local/lib/python2.7/dist-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.5.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.5.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth-1.4.1-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #236

[Apache Jenkins Server]

See 


--
[...truncated 10.12 KB...]
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5127

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4387

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3666

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1451

[Apache Jenkins Server]

See 


--
[...truncated 90.87 KB...]
'apache-beam-testing:bqjob_r4082f2247ab354f7_01620fd95605_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 12:18:46,285 b4e74e25 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 12:19:12,942 b4e74e25 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 12:19:15,567 b4e74e25 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.60s,  CPU:0.31s,  MaxMemory:25276kb 
STDOUT: Upload complete.
Waiting on bqjob_r39e037170f53f589_01620fd9c846_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r39e037170f53f589_01620fd9c846_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r39e037170f53f589_01620fd9c846_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 12:19:15,567 b4e74e25 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 12:19:30,779 b4e74e25 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 12:19:32,965 b4e74e25 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.17s,  CPU:0.24s,  MaxMemory:25588kb 
STDOUT: Upload complete.
Waiting on bqjob_r5cc0044ad71c384c_01620fda0c6c_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r5cc0044ad71c384c_01620fda0c6c_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r5cc0044ad71c384c_01620fda0c6c_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 12:19:32,966 b4e74e25 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 12:20:02,139 b4e74e25 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 12:20:04,561 b4e74e25 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.41s,  CPU:0.25s,  MaxMemory:25480kb 
STDOUT: Upload complete.
Waiting on bqjob_r7f5f98d6c9eb1fc1_01620fda86e1_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7f5f98d6c9eb1fc1_01620fda86e1_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5207

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3217) Add a performance test for HadoopInputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3217?focusedWorklogId=79207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79207
 ]

ASF GitHub Bot logged work on BEAM-3217:


Author: ASF GitHub Bot
Created on: 10/Mar/18 12:09
Start Date: 10/Mar/18 12:09
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #4758: [BEAM-3217] Jenkins 
job for HadoopInputFormatIOIT
URL: https://github.com/apache/beam/pull/4758#issuecomment-372025071
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79207)
Time Spent: 5h 50m  (was: 5h 40m)

> Add a performance test for HadoopInputFormatIO
> --
>
> Key: BEAM-3217
> URL: https://issues.apache.org/jira/browse/BEAM-3217
> Project: Beam
>  Issue Type: Test
>  Components: io-java-hadoop
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> We should add a large scale performance test for HadoopInputFormatIO. We 
> should use PerfKitBenchmarker based performance testing framework [1] to 
> manage Kubernetes based muti-node data store and to publish benchmark results.
> Example input format implementation to use: DBInputFormat to connect to a 
> Postgres instance.
> https://github.com/hanborq/hadoop/blob/master/src/mapred/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java
>  
> Example docker image to use: https://hub.docker.com/_/postgres/
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1007

[Apache Jenkins Server]

See 


--
[...truncated 1.11 KB...]
 > git rev-list --no-walk fc772c08f62960b6b944b07bc38df20a1054d3f9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1407891707490662407.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4695591666130389122.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8722924004043404685.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5711606418518386135.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5438310204953569872.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2890576885514630225.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #6

[Apache Jenkins Server]

See 


--
[...truncated 9.12 KB...]
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #235

[Apache Jenkins Server]

See 


--
[...truncated 10.12 KB...]
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #237

[Apache Jenkins Server]

See 


--
[...truncated 11.74 KB...]
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/usr/local/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.5.0.dev0)
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.5.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.5.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting proto-google-cloud-pubsub-v1==0.15.4 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.5.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/usr/local/lib/python2.7/dist-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.5.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.5.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth-1.4.1-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Collecting cachetools>=2.0.0 (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached cachetools-2.0.1-py2.py3-none-any.whl
Collecting future<0.17dev,>=0.16.0 (from 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5126

[Apache Jenkins Server]

See