date:20180123

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5741

2018-01-23 Thread Apache Jenkins Server

See

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5738

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Updated] (BEAM-3509) PARTITION BY in Beam SQL In Select Command

2018-01-23 Thread Kishan Kumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kishan Kumar updated BEAM-3509:
---
Fix Version/s: 2.3.0

> PARTITION BY in Beam SQL In Select Command
> --
>
> Key: BEAM-3509
> URL: https://issues.apache.org/jira/browse/BEAM-3509
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Assignee: Xu Mingmin
>Priority: Major
>  Labels: performance
> Fix For: 2.3.0
>
>
> Partition By Option Will Be Very Help Full for DataFlow Developer To Migrate 
> Query and Do Transformation on That because of Many *Netezza Query and Oracle 
> Query* Consists Of Partition By Which Makes SQL Query More Efficient. *The 
> alternative is Making Joins And Filtering It Can Be Done But It Makes Code 
> Unreadable And Performance Become bad for DataFlow Job.*
> Examples: SELECT MIN(COLUMN) OVER (PARTITION BY COLUMN NAME) FROM TABLENAME



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337026#comment-16337026
 ] 

ASF GitHub Bot commented on BEAM-3250:
--

bsidhom closed pull request #4418: [BEAM-3250] Migrate Flink and Spark 
ValidatesRunner to Gradle
URL: https://github.com/apache/beam/pull/4418
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/core/build.gradle b/sdks/java/core/build.gradle
index fae4638a1a1..913cd5ecb5b 100644
--- a/sdks/java/core/build.gradle
+++ b/sdks/java/core/build.gradle
@@ -16,6 +16,8 @@
  * limitations under the License.
  */
 
+import groovy.json.JsonOutput
+
 apply from: project(":").file("build_rules.gradle")
 applyJavaNature()
 applyAvroNature()
@@ -38,6 +40,14 @@ processResources {
   ]
 }
 
+configurations {
+flinkValidatesRunner
+sparkValidatesRunner {
+  // Testing the Spark runner causes a StackOverflowError if slf4j-jdk14 
is on the classpath
+  exclude group: "org.slf4j", module: "slf4j-jdk14"
+}
+}
+
 // Exclude tests that need a runner
 test {
   systemProperty "beamUseDummyRunner", "true"
@@ -71,6 +81,11 @@ dependencies {
   shadowTest library.java.slf4j_jdk14
   shadowTest library.java.mockito_core
   shadowTest "com.esotericsoftware.kryo:kryo:2.21"
+  flinkValidatesRunner project(path: project.path, configuration: "shadowTest")
+  flinkValidatesRunner project(path: ":runners:flink", configuration: "shadow")
+  sparkValidatesRunner project(path: project.path, configuration: "shadowTest")
+  sparkValidatesRunner project(path: ":runners:spark", configuration: "shadow")
+  sparkValidatesRunner project(path: ":runners:spark", configuration: 
"provided")
 }
 
 // Shade dependencies.
@@ -117,6 +132,110 @@ task packageTests(type: Jar) {
   classifier = "tests"
 }
 
+class ValidatesRunnerConfig {
+  // Runner name prefix
+  String runner
+  // List of test categories to exclude from this task. Optional.
+  List excludes
+  // Pipeline options command line arguments.
+  Map pipelineOptions
+  // Configuration to use for test runtime classpath.
+  FileCollection configuration
+  // Additional system properties to be set for tests. Optional.
+  Map systemProperties
+}
+
+def createValidatesRunner(Map m) {
+  def config = m as ValidatesRunnerConfig
+  assert config.runner != null
+  assert config.pipelineOptions != null
+  assert config.configuration != null
+  tasks.create(name: "${config.runner}ValidatesRunner", type: Test) {
+group = "Verification"
+description = "Validate ${config.runner} runner"
+def optionsList = config.pipelineOptions.collect {
+  def key = it.getKey()
+  def value = it.getValue()
+  "--${key}=${value}"
+}
+def pipelineOptions = JsonOutput.toJson(optionsList)
+
+systemProperty "beamTestPipelineOptions", pipelineOptions
+if (config.systemProperties) {
+  for (Map.Entry property : config.systemProperties) {
+systemProperty property.getKey(), property.getValue()
+  }
+}
+// TODO: Does Spark require a different forking strategy?
+maxParallelForks 4
+classpath = config.configuration
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  if (config.excludes) {
+excludeCategories(*config.excludes)
+  }
+}
+  }
+}
+
+def flinkExcludedCategories = [
+  'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders',
+  'org.apache.beam.sdk.testing.LargeKeys$Above100MB',
+  'org.apache.beam.sdk.testing.UsesSplittableParDo',
+  'org.apache.beam.sdk.testing.UsesCommittedMetrics',
+  'org.apache.beam.sdk.testing.UsesTestStream',
+]
+
+def validatesRunnerConfigs = [
+  [
+runner: "flinkBatch",
+excludes: flinkExcludedCategories,
+pipelineOptions: [
+  runner: "TestFlinkRunner",
+  streaming: false,
+],
+configuration: configurations.flinkValidatesRunner,
+  ],
+  [
+runner: "flinkStreaming",
+excludes: flinkExcludedCategories,
+pipelineOptions: [
+  runner: "TestFlinkRunner",
+  streaming: true,
+],
+configuration: configurations.flinkValidatesRunner,
+  ],
+  [
+runner: "sparkBatch",
+excludes: [
+  'org.apache.beam.sdk.testing.UsesSplittableParDo',
+  'org.apache.beam.sdk.testing.UsesCommittedMetrics',
+  'org.apache.beam.sdk.testing.UsesTestStream',
+  'org.apache.beam.sdk.testing.UsesCustomWindowMerging',
+],
+pipelineOptions: [
+  runner: "TestSparkRunner",
+  streaming: "false",
+  enableSparkMetricSinks: "false",
+],
+configuration: configurations.sparkValidatesRunner,
+systemProperties: [
+  "beam.spark.test.reuseSparkContext":

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3975

2018-01-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Python #828

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2503bad7511ef5f4856fc8af9c24b01a8561b3c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Commit message: "[BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)"
 > git rev-list 6663e6038ca4293ecb65627e30bca6e41556e6ff # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4234987051030053190.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1498415613070168989.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4972741531330052470.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins959434350845561699.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6433174149059762558.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3974

2018-01-23 Thread Apache Jenkins Server

See

[beam] branch master updated: Increment the Dataflow runner major version to 7.

2018-01-23 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 57e8525  Increment the Dataflow runner major version to 7.
57e8525 is described below

commit 57e8525a3c5514e10a746c32a5026505c05ca89a
Author: Marian Dvorsky 
AuthorDate: Tue Dec 5 15:50:52 2017 +0100

Increment the Dataflow runner major version to 7.
---
 runners/google-cloud-dataflow-java/build.gradle| 2 +-
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 sdks/python/apache_beam/runners/dataflow/internal/apiclient.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/runners/google-cloud-dataflow-java/build.gradle 
b/runners/google-cloud-dataflow-java/build.gradle
index 1656733..b0e2a89 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -34,7 +34,7 @@ evaluationDependsOn(":model:fn-execution")
 
 processResources {
   filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
-'dataflow.legacy_environment_major_version' : '6',
+'dataflow.legacy_environment_major_version' : '7',
 'dataflow.fnapi_environment_major_version' : '1',
 'dataflow.container_version' : 'beam-master-20180122'
   ]
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 8006ecc..27cf2e1 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -35,7 +35,7 @@
   
 
beam-master-20180122
 
1
-
6
+
7
   
 
   
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index dd6bf95..1cf80b7 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -52,7 +52,7 @@ from apache_beam.utils import retry
 # Environment version information. It is passed to the service during a
 # a job submission and is used by the service to establish what features
 # are expected by the workers.
-_LEGACY_ENVIRONMENT_MAJOR_VERSION = '6'
+_LEGACY_ENVIRONMENT_MAJOR_VERSION = '7'
 _FNAPI_ENVIRONMENT_MAJOR_VERSION = '1'
 
 

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Commented] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336985#comment-16336985
 ] 

ASF GitHub Bot commented on BEAM-3250:
--

lukecwik closed pull request #4436: [BEAM-3250] Migrate Flink ValidatesRunner 
to Gradle
URL: https://github.com/apache/beam/pull/4436
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index 2e8f281f886..4ca9558e4fd 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -16,6 +16,8 @@
  * limitations under the License.
  */
 
+import groovy.json.JsonOutput
+
 apply from: project(":").file("build_rules.gradle")
 applyJavaNature()
 
@@ -30,6 +32,7 @@ description = "Apache Beam :: Runners :: Flink"
  */
 evaluationDependsOn(":model:fn-execution")
 evaluationDependsOn(":runners:core-java")
+evaluationDependsOn(":sdks:java:core")
 
 test {
   systemProperty "log4j.configuration", "log4j-test.properties"
@@ -39,6 +42,10 @@ test {
   }
 }
 
+configurations {
+  validatesRunner
+}
+
 def flink_version = "1.4.0"
 
 dependencies {
@@ -57,19 +64,58 @@ dependencies {
   shadow "org.apache.flink:flink-java:$flink_version"
   shadow "org.apache.flink:flink-runtime_2.11:$flink_version"
   shadow "org.apache.flink:flink-streaming-java_2.11:$flink_version"
-  testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
-  testCompile project(":model:fn-execution").sourceSets.test.output
-  testCompile project(":runners:core-java").sourceSets.test.output
-  testCompile library.java.commons_lang3
-  testCompile library.java.hamcrest_core
-  testCompile library.java.junit
-  testCompile library.java.mockito_core
-  testCompile library.java.google_api_services_bigquery
-  testCompile library.java.jackson_dataformat_yaml
-  testCompile "org.apache.flink:flink-core:$flink_version:tests"
-  testCompile "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
-  testCompile "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
-  testCompile "org.apache.flink:flink-test-utils_2.11:$flink_version"
+  shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+  shadowTest project(":model:fn-execution").sourceSets.test.output
+  shadowTest project(":runners:core-java").sourceSets.test.output
+  shadowTest library.java.commons_lang3
+  shadowTest library.java.hamcrest_core
+  shadowTest library.java.junit
+  shadowTest library.java.mockito_core
+  shadowTest library.java.google_api_services_bigquery
+  shadowTest library.java.jackson_dataformat_yaml
+  shadowTest "org.apache.flink:flink-core:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-test-utils_2.11:$flink_version"
+  validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+  validatesRunner project(path: project.path, configuration: "shadow")
+}
+
+class ValidatesRunnerConfig {
+  String name
+  boolean streaming
+}
+
+def createValidatesRunnerTask(Map m) {
+  def config = m as ValidatesRunnerConfig
+  tasks.create(name: config.name, type: Test) {
+group = "Verification"
+def runnerType = config.streaming ? "streaming" : "batch"
+description = "Validates the ${runnerType} runner"
+def pipelineOptions = JsonOutput.toJson(["--runner=TestFlinkRunner", 
"--streaming=${config.streaming}"])
+systemProperty "beamTestPipelineOptions", pipelineOptions
+classpath = configurations.validatesRunner
+testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+maxParallelForks 4
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+  excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+}
+  }
+}
+
+createValidatesRunnerTask(name: "validatesRunnerBatch", streaming: false)
+createValidatesRunnerTask(name: "validatesRunnerStreaming", streaming: true)
+
+task validatesRunner {
+  group = "Verification"
+  description "Validates batch and streaming runners"
+  dependsOn validatesRunnerBatch
+  dependsOn validatesRunnerStreaming
 }
 
 task packageTests(type: Jar) {


 


This is an automated message from the Apache Git Service.
To respond to the

[beam] branch master updated: Migrate Flink ValidatesRunner to Gradle

2018-01-23 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 6affcb5  Migrate Flink ValidatesRunner to Gradle
6affcb5 is described below

commit 6affcb5b583ce261aa8d3af31500f2ee58d22e13
Author: Ben Sidhom 
AuthorDate: Wed Jan 17 16:28:13 2018 -0800

Migrate Flink ValidatesRunner to Gradle
---
 runners/flink/build.gradle | 72 +-
 1 file changed, 59 insertions(+), 13 deletions(-)

diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index 2e8f281..4ca9558 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -16,6 +16,8 @@
  * limitations under the License.
  */
 
+import groovy.json.JsonOutput
+
 apply from: project(":").file("build_rules.gradle")
 applyJavaNature()
 
@@ -30,6 +32,7 @@ description = "Apache Beam :: Runners :: Flink"
  */
 evaluationDependsOn(":model:fn-execution")
 evaluationDependsOn(":runners:core-java")
+evaluationDependsOn(":sdks:java:core")
 
 test {
   systemProperty "log4j.configuration", "log4j-test.properties"
@@ -39,6 +42,10 @@ test {
   }
 }
 
+configurations {
+  validatesRunner
+}
+
 def flink_version = "1.4.0"
 
 dependencies {
@@ -57,19 +64,58 @@ dependencies {
   shadow "org.apache.flink:flink-java:$flink_version"
   shadow "org.apache.flink:flink-runtime_2.11:$flink_version"
   shadow "org.apache.flink:flink-streaming-java_2.11:$flink_version"
-  testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
-  testCompile project(":model:fn-execution").sourceSets.test.output
-  testCompile project(":runners:core-java").sourceSets.test.output
-  testCompile library.java.commons_lang3
-  testCompile library.java.hamcrest_core
-  testCompile library.java.junit
-  testCompile library.java.mockito_core
-  testCompile library.java.google_api_services_bigquery
-  testCompile library.java.jackson_dataformat_yaml
-  testCompile "org.apache.flink:flink-core:$flink_version:tests"
-  testCompile "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
-  testCompile "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
-  testCompile "org.apache.flink:flink-test-utils_2.11:$flink_version"
+  shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+  shadowTest project(":model:fn-execution").sourceSets.test.output
+  shadowTest project(":runners:core-java").sourceSets.test.output
+  shadowTest library.java.commons_lang3
+  shadowTest library.java.hamcrest_core
+  shadowTest library.java.junit
+  shadowTest library.java.mockito_core
+  shadowTest library.java.google_api_services_bigquery
+  shadowTest library.java.jackson_dataformat_yaml
+  shadowTest "org.apache.flink:flink-core:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
+  shadowTest "org.apache.flink:flink-test-utils_2.11:$flink_version"
+  validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+  validatesRunner project(path: project.path, configuration: "shadow")
+}
+
+class ValidatesRunnerConfig {
+  String name
+  boolean streaming
+}
+
+def createValidatesRunnerTask(Map m) {
+  def config = m as ValidatesRunnerConfig
+  tasks.create(name: config.name, type: Test) {
+group = "Verification"
+def runnerType = config.streaming ? "streaming" : "batch"
+description = "Validates the ${runnerType} runner"
+def pipelineOptions = JsonOutput.toJson(["--runner=TestFlinkRunner", 
"--streaming=${config.streaming}"])
+systemProperty "beamTestPipelineOptions", pipelineOptions
+classpath = configurations.validatesRunner
+testClassesDirs = 
files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+maxParallelForks 4
+useJUnit {
+  includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+  excludeCategories 
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+  excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDo'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
+  excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+}
+  }
+}
+
+createValidatesRunnerTask(name: "validatesRunnerBatch", streaming: false)
+createValidatesRunnerTask(name: "validatesRunnerStreaming", streaming: true)
+
+task validatesRunner {
+  group = "Verification"
+  description "Validates batch and streaming runners"
+  dependsOn validatesRunnerBatch
+  dependsOn validatesRunnerStreaming
 }
 
 task packageTests(type: Jar) {

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #60

2018-01-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #52

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2503bad7511ef5f4856fc8af9c24b01a8561b3c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Commit message: "[BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)"
 > git rev-list 6663e6038ca4293ecb65627e30bca6e41556e6ff # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1289184226679926177.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6495969299854022623.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7332366641380954883.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7044802700758807005.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins220061205758015411.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7410287903677056664.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #53

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2503bad7511ef5f4856fc8af9c24b01a8561b3c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Commit message: "[BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)"
 > git rev-list 6663e6038ca4293ecb65627e30bca6e41556e6ff # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5725625044892819211.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6711472797529390797.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6851171667289824814.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2614255733180028882.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8202014674966302852.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8889721878531150224.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from

Build failed in Jenkins: beam_PerformanceTests_Spark #1272

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2503bad7511ef5f4856fc8af9c24b01a8561b3c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Commit message: "[BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)"
 > git rev-list 6663e6038ca4293ecb65627e30bca6e41556e6ff # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4646842280408110490.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4388617660501278483.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4970732402008683603.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6225989523351381683.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4206906398357499902.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #52

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2503bad7511ef5f4856fc8af9c24b01a8561b3c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Commit message: "[BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)"
 > git rev-list 6663e6038ca4293ecb65627e30bca6e41556e6ff # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8916842713776247461.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins100962239367213.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7819322353536314028.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3698901470406698134.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins581210951145288489.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r

[jira] [Resolved] (BEAM-3008) BigtableIO should use ValueProviders

2018-01-23 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3008.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> BigtableIO should use ValueProviders 
> -
>
> Key: BEAM-3008
> URL: https://issues.apache.org/jira/browse/BEAM-3008
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
> Fix For: 2.3.0
>
>
> [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO 
> templatization.  This Issue is a request to get a fully featured template for 
> BigtableIO.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-3008] Adds parameters templatization for Bigtable (#4357)

2018-01-23 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new f2503ba  [BEAM-3008] Adds parameters templatization for Bigtable 
(#4357)
f2503ba is described below

commit f2503bad7511ef5f4856fc8af9c24b01a8561b3c
Author: dmytroivanov4206 
AuthorDate: Wed Jan 24 06:51:31 2018 +0100

[BEAM-3008] Adds parameters templatization for Bigtable (#4357)
---
 .../beam/sdk/io/gcp/bigtable/BigtableConfig.java   | 100 
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   | 222 +++---
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |   4 +-
 .../sdk/io/gcp/bigtable/BigtableConfigTest.java| 252 +
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   | 104 +
 5 files changed, 494 insertions(+), 188 deletions(-)

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
index ba633d0..4d2e4ce 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
@@ -19,7 +19,6 @@
 package org.apache.beam.sdk.io.gcp.bigtable;
 
 import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.auto.value.AutoValue;
 import com.google.cloud.bigtable.config.BigtableOptions;
@@ -30,6 +29,7 @@ import java.io.Serializable;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 
@@ -43,19 +43,19 @@ abstract class BigtableConfig implements Serializable {
* Returns the project id being written to.
*/
   @Nullable
-  abstract String getProjectId();
+  abstract ValueProvider getProjectId();
 
   /**
* Returns the instance id being written to.
*/
   @Nullable
-  abstract String getInstanceId();
+  abstract ValueProvider getInstanceId();
 
   /**
* Returns the table being read from.
*/
   @Nullable
-  abstract String getTableId();
+  abstract ValueProvider getTableId();
 
   /**
* Returns the Google Cloud Bigtable instance being written to, and other 
parameters.
@@ -93,11 +93,11 @@ abstract class BigtableConfig implements Serializable {
   @AutoValue.Builder
   abstract static class Builder {
 
-abstract Builder setProjectId(String projectId);
+abstract Builder setProjectId(ValueProvider projectId);
 
-abstract Builder setInstanceId(String instanceId);
+abstract Builder setInstanceId(ValueProvider instanceId);
 
-abstract Builder setTableId(String tableId);
+abstract Builder setTableId(ValueProvider tableId);
 
 /**
  * @deprecated will be replaced by bigtable options configurator.
@@ -115,18 +115,18 @@ abstract class BigtableConfig implements Serializable {
 abstract BigtableConfig build();
   }
 
-  BigtableConfig withProjectId(String projectId) {
-checkNotNull(projectId, "Project Id of BigTable can not be null");
+  BigtableConfig withProjectId(ValueProvider projectId) {
+checkArgument(projectId != null, "Project Id of BigTable can not be null");
 return toBuilder().setProjectId(projectId).build();
   }
 
-  BigtableConfig withInstanceId(String instanceId) {
-checkNotNull(instanceId, "Instance Id of BigTable can not be null");
+  BigtableConfig withInstanceId(ValueProvider instanceId) {
+checkArgument(instanceId != null, "Instance Id of BigTable can not be 
null");
 return toBuilder().setInstanceId(instanceId).build();
   }
 
-  BigtableConfig withTableId(String tableId) {
-checkNotNull(tableId, "tableId can not be null");
+  BigtableConfig withTableId(ValueProvider tableId) {
+checkArgument(tableId != null, "tableId can not be null");
 return toBuilder().setTableId(tableId).build();
   }
 
@@ -135,13 +135,13 @@ abstract class BigtableConfig implements Serializable {
*/
   @Deprecated
   BigtableConfig withBigtableOptions(BigtableOptions options) {
-checkNotNull(options, "Bigtable options can not be null");
+checkArgument(options != null, "Bigtable options can not be null");
 return toBuilder().setBigtableOptions(options).build();
   }
 
   BigtableConfig withBigtableOptionsConfigurator(
 SerializableFunction 
configurator) {
-checkNotNull(configurator, "configurator can not be null");
+checkArgument(configurator != null,

[jira] [Commented] (BEAM-3524) Automate testing using python sdk container built at head

2018-01-23 Thread Alan Myrvold (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336886#comment-16336886
 ] 

Alan Myrvold commented on BEAM-3524:


https://github.com/apache/beam/pull/4476

> Automate testing using python sdk container built at head
> -
>
> Key: BEAM-3524
> URL: https://issues.apache.org/jira/browse/BEAM-3524
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-harness
>Reporter: Alan Myrvold
>Assignee: Robert Bradshaw
>Priority: Major
>
> As mentioned in BEAM-3411 we should have a test that builds a docker 
> container from sdks/python/container and runs an integration test with the 
> --worker_harness_container_image flag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #719

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-3412) Update BigTable client version to 1.0

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336828#comment-16336828
 ] 

ASF GitHub Bot commented on BEAM-3412:
--

sduskis closed pull request #4462: [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0
URL: https://github.com/apache/beam/pull/4462
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index e51ee28bd8e..10086b4824a 100644
--- a/build.gradle
+++ b/build.gradle
@@ -27,7 +27,7 @@ apply from: project(":").file("build_rules.gradle")
 // a dependency version which should match across multiple
 // Maven artifacts.
 def google_cloud_bigdataoss_version = "1.4.5"
-def bigtable_version = "1.0.0-pre3"
+def bigtable_version = "1.0.0"
 def google_clients_version = "1.22.0"
 def google_auth_version = "0.7.1"
 def grpc_version = "1.2.0"
diff --git a/pom.xml b/pom.xml
index fd65b2d9606..627551f68a8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -110,7 +110,7 @@
 2.33
 1.8.2
 v2-rev355-1.22.0
-1.0.0-pre3
+1.0.0
 v1-rev6-1.22.0
 0.1.18
 v2-rev8-1.22.0
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
index 06c459bb447..2206bdd86d4 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
@@ -18,8 +18,8 @@
 package org.apache.beam.sdk.io.gcp.bigtable;
 
 import com.google.bigtable.admin.v2.GetTableRequest;
-import com.google.bigtable.v2.MutateRowRequest;
 import com.google.bigtable.v2.MutateRowResponse;
+import com.google.bigtable.v2.MutateRowsRequest;
 import com.google.bigtable.v2.Mutation;
 import com.google.bigtable.v2.ReadRowsRequest;
 import com.google.bigtable.v2.Row;
@@ -30,9 +30,9 @@
 import com.google.cloud.bigtable.config.BigtableOptions;
 import com.google.cloud.bigtable.grpc.BigtableSession;
 import com.google.cloud.bigtable.grpc.BigtableTableName;
-import com.google.cloud.bigtable.grpc.async.AsyncExecutor;
 import com.google.cloud.bigtable.grpc.async.BulkMutation;
 import com.google.cloud.bigtable.grpc.scanner.ResultScanner;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.MoreObjects;
 import com.google.common.io.Closer;
 import com.google.common.util.concurrent.ListenableFuture;
@@ -94,13 +94,15 @@ public boolean tableExists(String tableId) throws 
IOException {
 }
   }
 
-  private class BigtableReaderImpl implements Reader {
+  @VisibleForTesting
+  static class BigtableReaderImpl implements Reader {
 private BigtableSession session;
 private final BigtableSource source;
 private ResultScanner results;
 private Row currentRow;
 
-public BigtableReaderImpl(BigtableSession session, BigtableSource source) {
+@VisibleForTesting
+BigtableReaderImpl(BigtableSession session, BigtableSource source) {
   this.session = session;
   this.source = source;
 }
@@ -119,7 +121,8 @@ public boolean start() throws IOException {
   ReadRowsRequest.Builder requestB =
   ReadRowsRequest.newBuilder()
   .setRows(rowSet)
-  
.setTableName(options.getInstanceName().toTableNameStr(source.getTableId()));
+  .setTableName(session.getOptions().getInstanceName()
+  .toTableNameStr(source.getTableId()));
   if (source.getRowFilter() != null) {
 requestB.setFilter(source.getRowFilter());
   }
@@ -166,17 +169,14 @@ public Row getCurrentRow() throws NoSuchElementException {
 }
   }
 
-  private static class BigtableWriterImpl implements Writer {
+  @VisibleForTesting
+  static class BigtableWriterImpl implements Writer {
 private BigtableSession session;
-private AsyncExecutor executor;
 private BulkMutation bulkMutation;
-private final String tableName;
 
-public BigtableWriterImpl(BigtableSession session, BigtableTableName 
tableName) {
+BigtableWriterImpl(BigtableSession session, BigtableTableName tableName) {
   this.session = session;
-  executor = session.createAsyncExecutor();
-  bulkMutation = session.createBulkMutation(tableName, executor);
-  this.tableName = tableName.toString();
+  bulkMutation = session.createBulkMutation(tableName);
 }
 
 @Override
@@ -189,7 +189,6 @@ public void flush() throws IOException {
   // We fail since flush() operation was interrupted.
   throw new IOException(e);
 }
-

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #5737

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Created] (BEAM-3524) Automate testing using python sdk container built at head

2018-01-23 Thread Alan Myrvold (JIRA)

Alan Myrvold created BEAM-3524:
--

 Summary: Automate testing using python sdk container built at head
 Key: BEAM-3524
 URL: https://issues.apache.org/jira/browse/BEAM-3524
 Project: Beam
  Issue Type: Test
  Components: sdk-py-harness
Reporter: Alan Myrvold
Assignee: Robert Bradshaw


As mentioned in BEAM-3411 we should have a test that builds a docker container 
from sdks/python/container and runs an integration test with the 
--worker_harness_container_image flag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

2018-01-23 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2345:
-

Assignee: (was: Jason Kuster)

> Version configuration of plugins / dependencies in root pom.xml is 
> inconsistent
> ---
>
> Key: BEAM-2345
> URL: https://issues.apache.org/jira/browse/BEAM-2345
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Priority: Minor
> Fix For: 2.3.0
>
>
> Versioning in root pom.xml in some places is controlled by the properties 
> section, sometimes is just inline. Move all versioning of plugins / 
> dependencies to properties section.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2345) Version configuration of plugins / dependencies in root pom.xml is inconsistent

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336767#comment-16336767
 ] 

ASF GitHub Bot commented on BEAM-2345:
--

aaltay closed pull request #3205: [BEAM-2345] Make versioning in root pom 
consistent.
URL: https://github.com/apache/beam/pull/3205
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pom.xml b/pom.xml
index 1bdaa97f525..1460891faf3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -100,7 +100,14 @@
 
 
 
-
+
+3.2
+
+
 1.14
 3.6
 1.1
@@ -109,13 +116,16 @@
 1.8.2
 v2-rev295-1.22.0
 0.9.7.1
-v1-rev6-1.22.0
-0.1.0
+1.6.8
 v2-rev8-1.22.0
+v1-rev6-1.22.0
 v1b3-rev198-1.20.0
 0.5.160222
 1.4.0
 1.3.0
+2.0.15
+3.0.1
+0.20.0
 1.0.0-rc2
 1.0-rc2
 1.4.1
@@ -124,9 +134,9 @@
 1.4.5
 1.0.2
 
0.5.160304
-20.0
 1.2.0
 
0.1.9
+20.0
 
+0.12
+
3.0.0
+4.3.0
+
1.0.0
+1.5.0
+1.0-beta-6
 2.0
-2.20
 2.20
+3.0.4
+0.7.8
+1.8
+3.0.0
+2.17
+3.0.0
 3.6.2
 3.0.1
+2.8.2
 3.0.0-M1
 1.6.0
 3.0.2
 3.0.0-M1
+2.5.3
 3.0.2
 3.0.0
+3.0.1
+1.5.0.Final
+0.5.0
+2.20
+2.3
 
+
 -Werror
 
-Xpkginfo:always
 nothing
-0.20.0
 
 
 kubectl
@@ -346,7 +376,7 @@
 
   org.eclipse.tycho
   tycho-compiler-jdt
-  0.26.0
+  ${tycho-compiler-jdt.version}
 
   
 
@@ -1085,7 +1115,7 @@
   
 com.google.errorprone
 error_prone_annotations
-2.0.15
+${error_prone_annotations.version}
   
 
   
@@ -1103,7 +1133,7 @@
   
 net.bytebuddy
 byte-buddy
-1.6.8
+${byte-buddy.version}
   
 
   
@@ -1261,24 +1291,24 @@
 
   org.xolstice.maven.plugins
   protobuf-maven-plugin
-  0.5.0
+  ${protobuf-maven-plugin.version}
 
 
 
   org.apache.maven.plugins
   maven-antrun-plugin
-  1.8
+  ${maven-antrun-plugin.version}
 
 
 
   org.apache.maven.plugins
   maven-checkstyle-plugin
-  2.17
+  ${maven-checkstyle-plugin.version}
   
 
   com.puppycrawl.tools
   checkstyle
-  6.19
+  ${puppycrawl.tools.checkstyle.version}
 
 
   org.apache.beam
@@ -1362,7 +1392,7 @@
 
   org.apache.maven.plugins
   maven-deploy-plugin
-  2.8.2
+  ${maven-deploy-plugin.version}
 
 
 
@@ -1391,7 +1421,7 @@
 
   org.apache.maven.plugins
   maven-source-plugin
-  3.0.1
+  ${maven-source-plugin.version}
 
 
 
@@ -1416,7 +1446,7 @@
 
   org.apache.rat
   apache-rat-plugin
-  0.12
+  ${apache-rat-plugin.version}
   
 
${project.build.directory}/${project.build.finalName}.rat
 false
@@ -1456,13 +1486,13 @@
 
   org.codehaus.mojo
   versions-maven-plugin
-  2.3
+  ${versions-maven-plugin.version}
 
 
 
   org.codehaus.mojo
   exec-maven-plugin
-  1.5.0
+  ${exec-maven-plugin.version}
   
 false
 
@@ -1478,7 +1508,7 @@
 
   org.jacoco
   jacoco-maven-plugin
-  0.7.8
+  ${jacoco-maven-plugin.version}
   
 
   
@@ -1508,7 +1538,7 @@
 
   org.eluder.coveralls
   coveralls-maven-plugin
-  4.3.0
+  ${coveralls-maven-plugin.version}
 
 
 
@@ -1545,7 +1575,7 @@
 
   org.eclipse.m2e
   lifecycle-mapping
-  1.0.0
+  ${eclipse.m2e.lifecycle-mapping.version}
   
 
   
@@ -1662,7 +1692,7 @@
 
   org.apache.maven.plugins
   maven-release-plugin
-  2.5.3
+  ${maven-release-plugin.version}
   
 release
 clean install
@@ -1676,7 +1706,7 @@
 
   org.apache.maven.plugins
   maven-assembly-plugin
-  3.0.0
+  ${maven-assembly-plugin.version}
   
 
   source-release-assembly
@@ -1690,7 +1720,7 @@
 
   org.codehaus.mojo
   findbugs-maven-plugin
-  3.0.4
+

[jira] [Commented] (BEAM-1267) BigQueryIO.Write should support the ignoreUnknownValues option

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336766#comment-16336766
 ] 

ASF GitHub Bot commented on BEAM-1267:
--

aaltay closed pull request #1778: [BEAM-1267] Adds ignoreUnknownValues option 
to BigQuery.Write
URL: https://github.com/apache/beam/pull/1778
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 4b19973f1bb..066827707f1 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -1641,7 +1641,7 @@ public static Bound withoutValidation() {
 
   // An option to indicate if table validation is desired. Default is true.
   final boolean validate;
-
+  @Nullable final Boolean ignoreUnknownValues;
   @Nullable private BigQueryServices bigQueryServices;
 
   private static class TranslateTableSpecFunction implements
@@ -1673,14 +1673,15 @@ public Bound() {
 CreateDisposition.CREATE_IF_NEEDED,
 WriteDisposition.WRITE_EMPTY,
 true /* validate */,
-null /* bigQueryServices */);
+null /* bigQueryServices */,
+false /* ignoreUnknownValues */);
   }
 
   private Bound(String name, @Nullable ValueProvider jsonTableRef,
   @Nullable SerializableFunction 
tableRefFunction,
   @Nullable ValueProvider jsonSchema,
   CreateDisposition createDisposition, WriteDisposition 
writeDisposition, boolean validate,
-  @Nullable BigQueryServices bigQueryServices) {
+  @Nullable BigQueryServices bigQueryServices, @Nullable Boolean 
ignoreUnknownValues) {
 super(name);
 this.jsonTableRef = jsonTableRef;
 this.tableRefFunction = tableRefFunction;
@@ -1689,6 +1690,7 @@ private Bound(String name, @Nullable 
ValueProvider jsonTableRef,
 this.writeDisposition = checkNotNull(writeDisposition, 
"writeDisposition");
 this.validate = validate;
 this.bigQueryServices = bigQueryServices;
+this.ignoreUnknownValues = ignoreUnknownValues;
   }
 
   /**
@@ -1730,7 +1732,7 @@ private Bound toTableRef(ValueProvider 
table) {
 return new Bound(name,
 NestedValueProvider.of(table, new TableRefToJson()),
 tableRefFunction, jsonSchema, createDisposition,
-writeDisposition, validate, bigQueryServices);
+writeDisposition, validate, bigQueryServices, ignoreUnknownValues);
   }
 
   /**
@@ -1759,7 +1761,7 @@ public Bound to(
   public Bound toTableReference(
   SerializableFunction 
tableRefFunction) {
 return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+writeDisposition, validate, bigQueryServices, ignoreUnknownValues);
   }
 
   /**
@@ -1771,7 +1773,7 @@ public Bound toTableReference(
   public Bound withSchema(TableSchema schema) {
 return new Bound(name, jsonTableRef, tableRefFunction,
 StaticValueProvider.of(toJsonString(schema)),
-createDisposition, writeDisposition, validate, bigQueryServices);
+createDisposition, writeDisposition, validate, bigQueryServices, 
ignoreUnknownValues);
   }
 
   /**
@@ -1780,7 +1782,7 @@ public Bound withSchema(TableSchema schema) {
   public Bound withSchema(ValueProvider schema) {
 return new Bound(name, jsonTableRef, tableRefFunction,
 NestedValueProvider.of(schema, new TableSchemaToJsonSchema()),
-createDisposition, writeDisposition, validate, bigQueryServices);
+createDisposition, writeDisposition, validate, bigQueryServices, 
ignoreUnknownValues);
   }
 
   /**
@@ -1790,7 +1792,7 @@ public Bound withSchema(ValueProvider 
schema) {
*/
   public Bound withCreateDisposition(CreateDisposition createDisposition) {
 return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
-createDisposition, writeDisposition, validate, bigQueryServices);
+createDisposition, writeDisposition, validate, bigQueryServices, 
ignoreUnknownValues);
   }
 
   /**
@@ -1800,7 +1802,7 @@ public Bound withCreateDisposition(CreateDisposition 
createDisposition) {
*/
   public

[jira] [Assigned] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

2018-01-23 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2273:
-

Assignee: (was: Jason Kuster)

> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2273) mvn clean doesn't fully clean up archetypes.

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336762#comment-16336762
 ] 

ASF GitHub Bot commented on BEAM-2273:
--

aaltay closed pull request #3107: [BEAM-2273] Fully clean up archetypes when 
running mvn clean
URL: https://github.com/apache/beam/pull/3107
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/maven-archetypes/pom.xml 
b/sdks/java/maven-archetypes/pom.xml
index b7fe2747daa..31263c321f4 100644
--- a/sdks/java/maven-archetypes/pom.xml
+++ b/sdks/java/maven-archetypes/pom.xml
@@ -69,7 +69,7 @@
 
   
 
-
+
 
   
@@ -87,6 +87,29 @@
   
 
   
+  
+maven-clean-plugin
+
+  
+
+  
examples/src/main/resources/archetype-resources
+  
+src/**/*
+src
+  
+  false
+
+
+  
examples-java8/src/main/resources/archetype-resources
+  
+src/**/*
+src
+  
+  false
+
+  
+
+  
 
   
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> mvn clean doesn't fully clean up archetypes.
> 
>
> Key: BEAM-2273
> URL: https://issues.apache.org/jira/browse/BEAM-2273
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>Priority: Major
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #718

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[robertwb] Remove legacy windowfn translation.

[ehudm] Pass PipelineOptions to FileSystem constructor.

--
[...truncated 182.81 KB...]
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0AJ%22H%0A%1Dref_Coder_GlobalWindowCoder_1%12%27%0A%25%0A%23%0A%21urn%3Abeam%3Acoders%3Aglobal_window%3A0.1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3973

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-3388) Reduce Go runtime reflective overhead

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336722#comment-16336722
 ] 

ASF GitHub Bot commented on BEAM-3388:
--

lostluck opened a new pull request #4474: [BEAM-3388] Use a typeswitch instead 
of reflect.Convert
URL: https://github.com/apache/beam/pull/4474
 
 
   Use a typeswitch instead of reflect.Convert when converting strings or bytes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reduce Go runtime reflective overhead
> -
>
> Key: BEAM-3388
> URL: https://issues.apache.org/jira/browse/BEAM-3388
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.3.0
>
>
> Go reflection is slow and we should avoid it in the Go SDK at runtime, when 
> possible -- especially on the fast paths. It seems unlikely that the language 
> runtime/libraries will improve any time soon: 
> https://github.com/golang/go/issues/7818.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2524) Update Google Cloud Console URL returned by DataflowRunner to support regions.

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336716#comment-16336716
 ] 

ASF GitHub Bot commented on BEAM-2524:
--

lostluck opened a new pull request #4473: [BEAM-2524] Update the gcloud cancel 
command to include the --region flag
URL: https://github.com/apache/beam/pull/4473
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update Google Cloud Console URL returned by DataflowRunner to support regions.
> --
>
> Key: BEAM-2524
> URL: https://issues.apache.org/jira/browse/BEAM-2524
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: 2.2.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Both the Java and Python Dataflow Runners need to be updated with a 
> regionalized form of the Google Cloud Console URL to support multiple 
> Dataflow Regions.
> The new URL format will be:
> https://console.cloud.corp.google.com/dataflow/jobsDetail/locations//jobs/?project=



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336697#comment-16336697
 ] 

ASF GitHub Bot commented on BEAM-3326:
--

tgroh opened a new pull request #4472: [BEAM-3326] Use a Data Service in 
SdkHarnessClient
URL: https://github.com/apache/beam/pull/4472
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #52

2018-01-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #51

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[github] Fix code comment to match code

[github] NotImplementedErrror --> NotImplementedError

[robertwb] Remove legacy windowfn translation.

[github] Update BUILD.md

[batbat] Added an example pipeline that uses stateful processing to output team

[robertwb] Manually specify direct runner for global-state modifying tests.

[robertwb] FakeSource requires direct runner.

[robertwb] Explicitly use DirectRunner in DirectRunner tests.

[altay] Disable combiner lifting optimization in DataflowRunner for fnapi

[ehudm] Pass PipelineOptions to FileSystem constructor.

[altay] Disable combiner lifting when only the streaming flag is set.

[tgroh] Register Environments in SdkComponents

[robertwb] Document DirectRunnerOnly tests.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6663e6038ca4293ecb65627e30bca6e41556e6ff (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6663e6038ca4293ecb65627e30bca6e41556e6ff
Commit message: "Merge pull request #4465 from udim/filesystem-options"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6698730620670428783.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins139150168316272.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2278614161297908123.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8415210076769696372.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins9149254646418380574.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #59

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[github] Fix code comment to match code

[github] NotImplementedErrror --> NotImplementedError

[robertwb] Remove legacy windowfn translation.

[github] Update BUILD.md

[batbat] Added an example pipeline that uses stateful processing to output team

[robertwb] Manually specify direct runner for global-state modifying tests.

[robertwb] FakeSource requires direct runner.

[robertwb] Explicitly use DirectRunner in DirectRunner tests.

[altay] Disable combiner lifting optimization in DataflowRunner for fnapi

[ehudm] Pass PipelineOptions to FileSystem constructor.

[altay] Disable combiner lifting when only the streaming flag is set.

[tgroh] Register Environments in SdkComponents

[robertwb] Document DirectRunnerOnly tests.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6663e6038ca4293ecb65627e30bca6e41556e6ff (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6663e6038ca4293ecb65627e30bca6e41556e6ff
Commit message: "Merge pull request #4465 from udim/filesystem-options"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins913125944226882558.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4103544688992317443.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8348227047984820155.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6982866687206506872.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5789913641502659075.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in

Build failed in Jenkins: beam_PerformanceTests_Spark #1271

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[github] Fix code comment to match code

[github] NotImplementedErrror --> NotImplementedError

[robertwb] Remove legacy windowfn translation.

[github] Update BUILD.md

[batbat] Added an example pipeline that uses stateful processing to output team

[robertwb] Manually specify direct runner for global-state modifying tests.

[robertwb] FakeSource requires direct runner.

[robertwb] Explicitly use DirectRunner in DirectRunner tests.

[altay] Disable combiner lifting optimization in DataflowRunner for fnapi

[ehudm] Pass PipelineOptions to FileSystem constructor.

[altay] Disable combiner lifting when only the streaming flag is set.

[tgroh] Register Environments in SdkComponents

[robertwb] Document DirectRunnerOnly tests.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6663e6038ca4293ecb65627e30bca6e41556e6ff (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6663e6038ca4293ecb65627e30bca6e41556e6ff
Commit message: "Merge pull request #4465 from udim/filesystem-options"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2461341295536553318.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5967001983519332829.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7767759344483771898.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7491816064975427158.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5721633099570889073.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in

Build failed in Jenkins: beam_PerformanceTests_Python #827

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[github] Fix code comment to match code

[github] NotImplementedErrror --> NotImplementedError

[robertwb] Remove legacy windowfn translation.

[github] Update BUILD.md

[batbat] Added an example pipeline that uses stateful processing to output team

[robertwb] Manually specify direct runner for global-state modifying tests.

[robertwb] FakeSource requires direct runner.

[robertwb] Explicitly use DirectRunner in DirectRunner tests.

[altay] Disable combiner lifting optimization in DataflowRunner for fnapi

[ehudm] Pass PipelineOptions to FileSystem constructor.

[altay] Disable combiner lifting when only the streaming flag is set.

[tgroh] Register Environments in SdkComponents

[robertwb] Document DirectRunnerOnly tests.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6663e6038ca4293ecb65627e30bca6e41556e6ff (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6663e6038ca4293ecb65627e30bca6e41556e6ff
Commit message: "Merge pull request #4465 from udim/filesystem-options"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5335031508465390845.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5320057679787662314.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9088297080184699183.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4743533065908698620.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8164324441801399250.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #51

2018-01-23 Thread Apache Jenkins Server

See 


Changes:

[github] Fix code comment to match code

[github] NotImplementedErrror --> NotImplementedError

[robertwb] Remove legacy windowfn translation.

[github] Update BUILD.md

[batbat] Added an example pipeline that uses stateful processing to output team

[robertwb] Manually specify direct runner for global-state modifying tests.

[robertwb] FakeSource requires direct runner.

[robertwb] Explicitly use DirectRunner in DirectRunner tests.

[altay] Disable combiner lifting optimization in DataflowRunner for fnapi

[ehudm] Pass PipelineOptions to FileSystem constructor.

[altay] Disable combiner lifting when only the streaming flag is set.

[tgroh] Register Environments in SdkComponents

[robertwb] Document DirectRunnerOnly tests.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6663e6038ca4293ecb65627e30bca6e41556e6ff (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6663e6038ca4293ecb65627e30bca6e41556e6ff
Commit message: "Merge pull request #4465 from udim/filesystem-options"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8619364744562380667.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6868156590967471696.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5846929301994466068.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7610595260248029401.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5690227376243413777.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3972

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Resolved] (BEAM-2804) support TIMESTAMP in Sort

2018-01-23 Thread Robert Bradshaw (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw resolved BEAM-2804.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> support TIMESTAMP in Sort
> -
>
> Key: BEAM-2804
> URL: https://issues.apache.org/jira/browse/BEAM-2804
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Shayang Zang
>Priority: Minor
>  Labels: beginner
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3099) Implement HDFS FileSystem for Python SDK

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1633#comment-1633
 ] 

ASF GitHub Bot commented on BEAM-3099:
--

udim opened a new pull request #4471: [BEAM-3099] Split out BufferedReader and 
BufferedWriter from gcsio.
URL: https://github.com/apache/beam/pull/4471
 
 
   Most of the code in filesystemio.py is copied verbatim from gcsio.py.
   The Downloader and Uploader classes are new.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Implement HDFS FileSystem for Python SDK
> 
>
> Key: BEAM-3099
> URL: https://issues.apache.org/jira/browse/BEAM-3099
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Udi Meiri
>Priority: Major
>
> Currently Java SDK has HDFS support but Python SDK does not. With current 
> portability efforts other runners may soon be able to use Python SDK. Having 
> HDFS support will allow these runners to execute large scale jobs without 
> using GCS. 
> Following suggests some libraries that can be used to connect to HDFS from 
> Python.
> http://wesmckinney.com/blog/python-hdfs-interfaces/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3523) Attach Python logging messages to namespaced logger

2018-01-23 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336655#comment-16336655
 ] 

Ahmet Altay commented on BEAM-3523:
---

> What is your opinion on how involved I should be in this? I don't claim to 
>understand the full architecture behind the python SDK, but I'd love to help 
>in anyway I can in a way that won't cause any undue delay.

If you are interested, you can start planning a doc or a PR. Otherwise I do not 
think we will work on this anytime soon.

> Attach Python logging messages to namespaced logger
> ---
>
> Key: BEAM-3523
> URL: https://issues.apache.org/jira/browse/BEAM-3523
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Alex Milstead
>Assignee: Ahmet Altay
>Priority: Minor
>
> The python SDK currently uses {{logging.(info|error|debug|etc)}} for log 
> messages. This can be disruptive or unexpected when integrating the SDK into 
> existing applications.
> I would like to request updating the SDK to enforce automatic module based 
> namespaces in python code (i.e. {{logger = logging.getLogger(__name__)}}) so 
> that all {{apache_beam}} output can be controlled by an integrating 
> application without the need to modify the root logging configuration.
> I'd be happy to submit a PR for this myself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3099) Implement HDFS FileSystem for Python SDK

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336649#comment-16336649
 ] 

ASF GitHub Bot commented on BEAM-3099:
--

chamikaramj closed pull request #4465: [BEAM-3099] Pass PipelineOptions to 
FileSystem constructor.
URL: https://github.com/apache/beam/pull/4465
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/filesystem.py 
b/sdks/python/apache_beam/io/filesystem.py
index 0efdb0ef751..424462ab30c 100644
--- a/sdks/python/apache_beam/io/filesystem.py
+++ b/sdks/python/apache_beam/io/filesystem.py
@@ -431,6 +431,12 @@ class FileSystem(BeamPlugin):
   __metaclass__ = abc.ABCMeta
   CHUNK_SIZE = 1  # Chuck size in the batch operations
 
+  def __init__(self, pipeline_options):
+"""
+Args:
+  pipeline_options: Instance of ``PipelineOptions``.
+"""
+
   @staticmethod
   def _get_compression_type(path, compression_type):
 if compression_type == CompressionTypes.AUTO:
diff --git a/sdks/python/apache_beam/io/filesystems.py 
b/sdks/python/apache_beam/io/filesystems.py
index 0c82a7e25dc..dad4e5f9f27 100644
--- a/sdks/python/apache_beam/io/filesystems.py
+++ b/sdks/python/apache_beam/io/filesystems.py
@@ -42,6 +42,17 @@ class FileSystems(object):
   """
   URI_SCHEMA_PATTERN = re.compile('(?P[a-zA-Z][-a-zA-Z0-9+.]*)://.*')
 
+  _pipeline_options = None
+
+  @classmethod
+  def set_options(cls, pipeline_options):
+"""Set filesystem options.
+
+Args:
+  pipeline_options: Instance of ``PipelineOptions``.
+"""
+cls._options = pipeline_options
+
   @staticmethod
   def get_scheme(path):
 match_result = FileSystems.URI_SCHEMA_PATTERN.match(path.strip())
@@ -60,7 +71,7 @@ def get_filesystem(path):
   if len(systems) == 0:
 raise ValueError('Unable to get the Filesystem for path %s' % path)
   elif len(systems) == 1:
-return systems[0]()
+return systems[0](pipeline_options=FileSystems._pipeline_options)
   else:
 raise ValueError('Found more than one filesystem for path %s' % path)
 except ValueError:
diff --git a/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py 
b/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
index c174e48778e..bc55b08f7dd 100644
--- a/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py
@@ -24,6 +24,7 @@
 
 from apache_beam.io.filesystem import BeamIOError
 from apache_beam.io.filesystem import FileMetadata
+from apache_beam.options.pipeline_options import PipelineOptions
 
 # Protect against environments where apitools library is not available.
 # pylint: disable=wrong-import-order, wrong-import-position
@@ -37,39 +38,40 @@
 @unittest.skipIf(gcsfilesystem is None, 'GCP dependencies are not installed')
 class GCSFileSystemTest(unittest.TestCase):
 
+  def setUp(self):
+pipeline_options = PipelineOptions()
+self.fs = gcsfilesystem.GCSFileSystem(pipeline_options=pipeline_options)
+
   def test_scheme(self):
-file_system = gcsfilesystem.GCSFileSystem()
-self.assertEqual(file_system.scheme(), 'gs')
+self.assertEqual(self.fs.scheme(), 'gs')
 self.assertEqual(gcsfilesystem.GCSFileSystem.scheme(), 'gs')
 
   def test_join(self):
-file_system = gcsfilesystem.GCSFileSystem()
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path', 'to', 'file'))
+ self.fs.join('gs://bucket/path', 'to', 'file'))
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path', 'to/file'))
+ self.fs.join('gs://bucket/path', 'to/file'))
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path', '/to/file'))
+ self.fs.join('gs://bucket/path', '/to/file'))
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path/', 'to', 'file'))
+ self.fs.join('gs://bucket/path/', 'to', 'file'))
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path/', 'to/file'))
+ self.fs.join('gs://bucket/path/', 'to/file'))
 self.assertEqual('gs://bucket/path/to/file',
- file_system.join('gs://bucket/path/', '/to/file'))
+ self.fs.join('gs://bucket/path/', '/to/file'))
 with self.assertRaises(ValueError):
-  file_system.join('/bucket/path/', '/to/file')
+  self.fs.join('/bucket/path/', '/to/file')
 
   def test_split(self):
-file_system = gcsfilesystem.GCSFileSystem()

[beam] branch master updated (d682042 -> 6663e60)

2018-01-23 Thread chamikara

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d682042  Merge pull request #4212 Remove legacy windowfn translation.
 add e8a725e  Pass PipelineOptions to FileSystem constructor.
 new 6663e60  Merge pull request #4465 from udim/filesystem-options

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filesystem.py   |  6 ++
 sdks/python/apache_beam/io/filesystems.py  | 13 +++-
 .../apache_beam/io/gcp/gcsfilesystem_test.py   | 71 +-
 sdks/python/apache_beam/io/hadoopfilesystem.py |  4 +-
 .../python/apache_beam/io/hadoopfilesystem_test.py |  4 +-
 sdks/python/apache_beam/io/localfilesystem_test.py |  4 +-
 sdks/python/apache_beam/pipeline.py|  3 +
 7 files changed, 59 insertions(+), 46 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] 01/01: Merge pull request #4465 from udim/filesystem-options

2018-01-23 Thread chamikara

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6663e6038ca4293ecb65627e30bca6e41556e6ff
Merge: d682042 e8a725e
Author: Chamikara Jayalath 
AuthorDate: Tue Jan 23 16:40:14 2018 -0800

Merge pull request #4465 from udim/filesystem-options

[BEAM-3099] Pass PipelineOptions to FileSystem constructor.

 sdks/python/apache_beam/io/filesystem.py   |  6 ++
 sdks/python/apache_beam/io/filesystems.py  | 13 +++-
 .../apache_beam/io/gcp/gcsfilesystem_test.py   | 71 +-
 sdks/python/apache_beam/io/hadoopfilesystem.py |  4 +-
 .../python/apache_beam/io/hadoopfilesystem_test.py |  4 +-
 sdks/python/apache_beam/io/localfilesystem_test.py |  4 +-
 sdks/python/apache_beam/pipeline.py|  3 +
 7 files changed, 59 insertions(+), 46 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Commented] (BEAM-2524) Update Google Cloud Console URL returned by DataflowRunner to support regions.

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336632#comment-16336632
 ] 

ASF GitHub Bot commented on BEAM-2524:
--

lostluck closed pull request #4144: [BEAM-2524] Update the gcloud cancel 
command to include the --region flag.
URL: https://github.com/apache/beam/pull/4144
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index cf464066c12..3388c52dfa2 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -214,8 +214,8 @@ public static String 
getGcloudCancelCommand(DataflowPipelineOptions options, Str
 }
 
 // Assemble cancel command from optional prefix and project/job parameters.
-return String.format("%s%s jobs --project=%s cancel %s",
-dataflowApiOverridePrefix, GCLOUD_DATAFLOW_PREFIX, 
options.getProject(), jobId);
+return String.format("%s%s jobs --project=%s cancel %s --region=%s",
+dataflowApiOverridePrefix, GCLOUD_DATAFLOW_PREFIX, 
options.getProject(), jobId, options.getRegion());
   }
 
   public static State toState(String stateName) {
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
index 499198221a7..1996e6bd5a5 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
@@ -49,6 +49,7 @@
 @RunWith(JUnit4.class)
 public class MonitoringUtilTest {
   private static final String PROJECT_ID = "someProject";
+  private static final String REGION_ID = "thatRegion";
   private static final String JOB_ID = "1234";
 
   @Rule public ExpectedLogs expectedLogs = 
ExpectedLogs.none(LoggingHandler.class);
@@ -119,9 +120,10 @@ public void testDontOverrideEndpointWithDefaultApi() {
 DataflowPipelineOptions options =
 PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
 options.setProject(PROJECT_ID);
+options.setRegion(REGION_ID);
 options.setGcpCredential(new TestCredential());
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
-assertEquals("gcloud dataflow jobs --project=someProject cancel 1234", 
cancelCommand);
+assertEquals("gcloud dataflow jobs --project=someProject cancel 1234 
--region=thatRegion", cancelCommand);
   }
 
   @Test
@@ -129,13 +131,14 @@ public void 
testOverridesEndpointWithStagedDataflowEndpoint() {
 DataflowPipelineOptions options =
 PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
 options.setProject(PROJECT_ID);
+options.setRegion(REGION_ID);
 options.setGcpCredential(new TestCredential());
 String stagingDataflowEndpoint = "v0neverExisted";
 options.setDataflowEndpoint(stagingDataflowEndpoint);
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
 assertEquals(
 
"CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW=https://dataflow.googleapis.com/v0neverExisted/
 "
-+ "gcloud dataflow jobs --project=someProject cancel 1234",
++ "gcloud dataflow jobs --project=someProject cancel 1234 
--region=thatRegion",
 cancelCommand);
   }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update Google Cloud Console URL returned by DataflowRunner to support regions.
> --
>
> Key: BEAM-2524
> URL: https://issues.apache.org/jira/browse/BEAM-2524
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: 2.2.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Both the Java and Python Dataflow Runners need to be

[beam] branch master updated (3920cd7 -> d682042)

2018-01-23 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3920cd7  Merge pull request #4454 [BEAM-3490] Explicitly use direct 
runner in non-portable tests.
 add 9b10739  Remove legacy windowfn translation.
 new d682042  Merge pull request #4212 Remove legacy windowfn translation.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/core/construction/WindowingStrategyTranslation.java| 2 --
 1 file changed, 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4212 Remove legacy windowfn translation.

2018-01-23 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d682042741079226af3ce2bf87a7a4452a2c9ba5
Merge: 3920cd7 9b10739
Author: Robert Bradshaw 
AuthorDate: Tue Jan 23 16:23:03 2018 -0800

Merge pull request #4212 Remove legacy windowfn translation.

 .../beam/runners/core/construction/WindowingStrategyTranslation.java| 2 --
 1 file changed, 2 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests.

2018-01-23 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3920cd7b1270f658b79480b61ac15766d38f
Merge: 19663de 4f92495
Author: Robert Bradshaw 
AuthorDate: Tue Jan 23 16:13:53 2018 -0800

Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in 
non-portable tests.

 sdks/python/apache_beam/pipeline_test.py  | 15 +--
 .../apache_beam/runners/dataflow/native_io/iobase_test.py |  3 ++-
 .../apache_beam/runners/direct/direct_runner_test.py  |  2 +-
 .../apache_beam/runners/direct/sdf_direct_runner_test.py  |  6 +++---
 4 files changed, 15 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Commented] (BEAM-3490) Reasonable Python direct runner batch performance.

2018-01-23 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336611#comment-16336611
 ] 

ASF GitHub Bot commented on BEAM-3490:
--

robertwb closed pull request #4454: [BEAM-3490] Explicitly use direct runner in 
non-portable tests.
URL: https://github.com/apache/beam/pull/4454
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index 22f89721ef5..634483e4160 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -160,8 +160,9 @@ def test_create_singleton_pcollection(self):
 
   # TODO(BEAM-1555): Test is failing on the service, with FakeSource.
   # @attr('ValidatesRunner')
-  def test_metrics_in_source(self):
-pipeline = TestPipeline()
+  def test_metrics_in_fake_source(self):
+# FakeSource mock requires DirectRunner.
+pipeline = TestPipeline(runner='DirectRunner')
 pcoll = pipeline | Read(FakeSource([1, 2, 3, 4, 5, 6]))
 assert_that(pcoll, equal_to([1, 2, 3, 4, 5, 6]))
 res = pipeline.run()
@@ -171,8 +172,9 @@ def test_metrics_in_source(self):
 self.assertEqual(outputs_counter.key.metric.name, 'outputs')
 self.assertEqual(outputs_counter.committed, 6)
 
-  def test_read(self):
-pipeline = TestPipeline()
+  def test_fake_read(self):
+# FakeSource mock requires DirectRunner.
+pipeline = TestPipeline(runner='DirectRunner')
 pcoll = pipeline | 'read' >> Read(FakeSource([1, 2, 3]))
 assert_that(pcoll, equal_to([1, 2, 3]))
 pipeline.run()
@@ -326,7 +328,8 @@ def get_overrides():
 
 file_system_override_mock.side_effect = get_overrides
 
-with Pipeline() as p:
+# Specify DirectRunner as it's the one patched above.
+with Pipeline(runner='DirectRunner') as p:
   pcoll = p | beam.Create([1, 2, 3]) | 'Multiply' >> DoubleParDo()
   assert_that(pcoll, equal_to([3, 6, 9]))
 
@@ -512,7 +515,7 @@ def __reduce__(self):
 class DirectRunnerRetryTests(unittest.TestCase):
 
   def test_retry_fork_graph(self):
-p = beam.Pipeline()
+p = beam.Pipeline(runner='DirectRunner')
 
 # TODO(mariagh): Remove the use of globals from the test.
 global count_b, count_c # pylint: disable=global-variable-undefined
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/iobase_test.py 
b/sdks/python/apache_beam/runners/dataflow/native_io/iobase_test.py
index 01fd35f9cf9..ce772572874 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/iobase_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/iobase_test.py
@@ -182,7 +182,8 @@ def __exit__(self, *unused_args):
   def Write(self, value):
 self.written_values.append(value)
 
-p = TestPipeline()
+# Records in-memory writes, only works on Direct runner.
+p = TestPipeline(runner='DirectRunner')
 sink = FakeSink()
 p | Create(['a', 'b', 'c']) | _NativeWrite(sink)  # pylint: 
disable=expression-not-assigned
 p.run()
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner_test.py 
b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
index 1c8b7855aad..1b51d05aae0 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
@@ -27,7 +27,7 @@ class DirectPipelineResultTest(unittest.TestCase):
   def test_waiting_on_result_stops_executor_threads(self):
 pre_test_threads = set(t.ident for t in threading.enumerate())
 
-pipeline = test_pipeline.TestPipeline()
+pipeline = test_pipeline.TestPipeline(runner='DirectRunner')
 _ = (pipeline | beam.Create([{'foo': 'bar'}]))
 result = pipeline.run()
 result.wait_until_finish()
diff --git a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py 
b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
index 7ab6dde9397..c1df7da52c4 100644
--- a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
@@ -145,7 +145,7 @@ def run_sdf_read_pipeline(
 
 assert len(expected_data) > 0
 
-with TestPipeline() as p:
+with TestPipeline(runner='DirectRunner') as p:
   pc1 = (p
  | 'Create1' >> beam.Create(file_names)
  | 'SDF' >> beam.ParDo(ReadFiles(resume_count)))
@@ -205,7 +205,7 @@ def test_sdf_with_resume_multiple_elements(self):
 resume_count)
 
   def test_sdf_with_windowed_timestamped_input(self):
-with TestPipeline() as p:
+with TestPipeline(runner='DirectRunner') as p:
   result = (p
 | beam.Create([1, 3, 5, 10])
 |

[beam] branch master updated (19663de -> 3920cd7)

2018-01-23 Thread robertwb

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 19663de  Merge pull request #4278 from itamaro/patch-1
 add 9641567  Manually specify direct runner for global-state modifying 
tests.
 add ade4fbd  FakeSource requires direct runner.
 add 194c393  Explicitly use DirectRunner in DirectRunner tests.
 add 4f92495  Document DirectRunnerOnly tests.
 new 3920cd7  Merge pull request #4454 [BEAM-3490] Explicitly use direct 
runner in non-portable tests.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline_test.py  | 15 +--
 .../apache_beam/runners/dataflow/native_io/iobase_test.py |  3 ++-
 .../apache_beam/runners/direct/direct_runner_test.py  |  2 +-
 .../apache_beam/runners/direct/sdf_direct_runner_test.py  |  6 +++---
 4 files changed, 15 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Commented] (BEAM-3523) Attach Python logging messages to namespaced logger

2018-01-23 Thread Alex Milstead (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336598#comment-16336598
 ] 

Alex Milstead commented on BEAM-3523:
-

[~altay] Thanks for the quick response!
{quote}Perhaps we can consider using a single name space for the whole beam sdk 
rather than module based logging.
{quote}
I think that is 100% acceptable way to do this. Module naming is easy, but as 
you've pointed out here, the real ask is giving it some kind of namespace 
regardless.

What is your opinion on how involved I should be in this? I don't claim to 
understand the full architecture behind the python SDK, but I'd love to help in 
anyway I can in a way that won't cause any undue delay.

> Attach Python logging messages to namespaced logger
> ---
>
> Key: BEAM-3523
> URL: https://issues.apache.org/jira/browse/BEAM-3523
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Alex Milstead
>Assignee: Ahmet Altay
>Priority: Minor
>
> The python SDK currently uses {{logging.(info|error|debug|etc)}} for log 
> messages. This can be disruptive or unexpected when integrating the SDK into 
> existing applications.
> I would like to request updating the SDK to enforce automatic module based 
> namespaces in python code (i.e. {{logger = logging.getLogger(__name__)}}) so 
> that all {{apache_beam}} output can be controlled by an integrating 
> application without the need to modify the root logging configuration.
> I'd be happy to submit a PR for this myself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3523) Attach Python logging messages to namespaced logger

2018-01-23 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336588#comment-16336588
 ] 

Ahmet Altay commented on BEAM-3523:
---

[~amilstead] thank you for the idea. I think this will be a positive move 
however there will be some complications. For example workers assume that the 
root logger will be used for logging 
([https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/logger.py).]
 And we also need to establish tests to see that this works as expected for all 
runners.

Perhaps we can consider using a single name space for the whole beam sdk rather 
than module based logging.

> Attach Python logging messages to namespaced logger
> ---
>
> Key: BEAM-3523
> URL: https://issues.apache.org/jira/browse/BEAM-3523
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Alex Milstead
>Assignee: Ahmet Altay
>Priority: Minor
>
> The python SDK currently uses {{logging.(info|error|debug|etc)}} for log 
> messages. This can be disruptive or unexpected when integrating the SDK into 
> existing applications.
> I would like to request updating the SDK to enforce automatic module based 
> namespaces in python code (i.e. {{logger = logging.getLogger(__name__)}}) so 
> that all {{apache_beam}} output can be controlled by an integrating 
> application without the need to modify the root logging configuration.
> I'd be happy to submit a PR for this myself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (584292e -> 19663de)

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 584292e  Merge pull request #4359 from cclauss/patch-1
 add ab85e0c  Fix code comment to match code
 new 19663de  Merge pull request #4278 from itamaro/patch-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/src/main/java/org/apache/beam/examples/MinimalWordCount.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4278 from itamaro/patch-1

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 19663de8ea0278874b2437f8ac2888a9f1363231
Merge: 584292e ab85e0c
Author: Ahmet Altay 
AuthorDate: Tue Jan 23 15:21:46 2018 -0800

Merge pull request #4278 from itamaro/patch-1

Fix code comment to match code

 .../java/src/main/java/org/apache/beam/examples/MinimalWordCount.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Created] (BEAM-3523) Attach Python logging messages to namespaced logger

2018-01-23 Thread Alex Milstead (JIRA)

Alex Milstead created BEAM-3523:
---

 Summary: Attach Python logging messages to namespaced logger
 Key: BEAM-3523
 URL: https://issues.apache.org/jira/browse/BEAM-3523
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Alex Milstead
Assignee: Ahmet Altay


The python SDK currently uses {{logging.(info|error|debug|etc)}} for log 
messages. This can be disruptive or unexpected when integrating the SDK into 
existing applications.

I would like to request updating the SDK to enforce automatic module based 
namespaces in python code (i.e. {{logger = logging.getLogger(__name__)}}) so 
that all {{apache_beam}} output can be controlled by an integrating application 
without the need to modify the root logging configuration.

I'd be happy to submit a PR for this myself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (02e9a96 -> 584292e)

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 02e9a96  Merge pull request #4399 from pshevtsov/patch-1
 add 23ba6ba  NotImplementedErrror --> NotImplementedError
 new 584292e  Merge pull request #4359 from cclauss/patch-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/worker/sdk_worker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (a16abdf -> 02e9a96)

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a16abdf  Merge pull request #4464
 add 1057d9a  Update BUILD.md
 new 02e9a96  Merge pull request #4399 from pshevtsov/patch-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/BUILD.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4359 from cclauss/patch-1

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 584292e4ac28eda689c51de5f99670266bf7ce09
Merge: 02e9a96 23ba6ba
Author: Ahmet Altay 
AuthorDate: Tue Jan 23 15:13:12 2018 -0800

Merge pull request #4359 from cclauss/patch-1

NotImplementedErrror --> NotImplementedError

 sdks/python/apache_beam/runners/worker/sdk_worker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4399 from pshevtsov/patch-1

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 02e9a960e05e6509a0dcb52cbc88fcbb3d75cae9
Merge: a16abdf 1057d9a
Author: Ahmet Altay 
AuthorDate: Tue Jan 23 15:12:28 2018 -0800

Merge pull request #4399 from pshevtsov/patch-1

Minor correction in BUILD.md

 sdks/go/BUILD.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Commented] (BEAM-3441) Allow ValueProvider for JdbcIO.DataSourceConfiguration

2018-01-23 Thread Chamikara Jayalath (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336473#comment-16336473
 ] 

Chamikara Jayalath commented on BEAM-3441:
--

Thanks. I'll take a look.

 

 

> Allow ValueProvider for JdbcIO.DataSourceConfiguration
> --
>
> Key: BEAM-3441
> URL: https://issues.apache.org/jira/browse/BEAM-3441
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Sameer Abhyankar
>Assignee: Sameer Abhyankar
>Priority: Major
>
> Currently JdbcIO only supports ValueProviders for queries but not for the 
> DataSourceConfiguration itself (i.e. driverClassName, url, username, password 
> etc.) These should support ValueProviders to allow the use of JdbcIO in 
> templates.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3441) Allow ValueProvider for JdbcIO.DataSourceConfiguration

2018-01-23 Thread Sameer Abhyankar (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336462#comment-16336462
 ] 

Sameer Abhyankar commented on BEAM-3441:


Hi [~chamikara] - Can you review this PR?

> Allow ValueProvider for JdbcIO.DataSourceConfiguration
> --
>
> Key: BEAM-3441
> URL: https://issues.apache.org/jira/browse/BEAM-3441
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Sameer Abhyankar
>Assignee: Sameer Abhyankar
>Priority: Major
>
> Currently JdbcIO only supports ValueProviders for queries but not for the 
> DataSourceConfiguration itself (i.e. driverClassName, url, username, password 
> etc.) These should support ValueProviders to allow the use of JdbcIO in 
> templates.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3971

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-3342) Create a Cloud Bigtable Python connector

2018-01-23 Thread Solomon Duskis (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336429#comment-16336429
 ] 

Solomon Duskis commented on BEAM-3342:
--

It turns out that we have quite a bit of work to do on the core Cloud Bigtable 
python client in order to make an effective Beam connector.  It could be a 
while before the client is ready.  

> Create a Cloud Bigtable Python connector
> 
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
>
> I would like to create a Cloud Bigtable python connector.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3160) Type based coder inference incorrectly assumes that a coder for one type is equivalent to every other coder for that type.

2018-01-23 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3160:
---

Assignee: Luke Cwik

> Type based coder inference incorrectly assumes that a coder for one type is 
> equivalent to every other coder for that type.
> --
>
> Key: BEAM-3160
> URL: https://issues.apache.org/jira/browse/BEAM-3160
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.3.0
>
>
> We should prevent coder inference from assuming that two coders for the same 
> type are interchangeable.
> Two Avro GenericRecord coders with different schemas are considered identical 
> and an arbitrary one is returned by the Coder/Type inference system if the 
> GenericRecord type appears multiple times.
> e.g.
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaA)), 
> IterableCoder.of(AvroCoder.of(SchemaB)))* after coder inference for the type 
> *KV* will return 
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaX)), 
> IterableCoder.of(AvroCoder.of(SchemaX)))* where SchemaX is either SchemaA or 
> SchemaB.
> Code:
> https://github.com/apache/beam/blob/v2.1.1/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L420
>  and other Type -> Coder maps in the same file should prevent insertion if 
> the type already exists and the coders aren't equal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3160) Type based coder inference incorrectly assumes that a coder for one type is equivalent to every other coder for that type.

2018-01-23 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3160:

Priority: Blocker  (was: Major)

> Type based coder inference incorrectly assumes that a coder for one type is 
> equivalent to every other coder for that type.
> --
>
> Key: BEAM-3160
> URL: https://issues.apache.org/jira/browse/BEAM-3160
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Priority: Blocker
> Fix For: 2.3.0
>
>
> We should prevent coder inference from assuming that two coders for the same 
> type are interchangeable.
> Two Avro GenericRecord coders with different schemas are considered identical 
> and an arbitrary one is returned by the Coder/Type inference system if the 
> GenericRecord type appears multiple times.
> e.g.
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaA)), 
> IterableCoder.of(AvroCoder.of(SchemaB)))* after coder inference for the type 
> *KV* will return 
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaX)), 
> IterableCoder.of(AvroCoder.of(SchemaX)))* where SchemaX is either SchemaA or 
> SchemaB.
> Code:
> https://github.com/apache/beam/blob/v2.1.1/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L420
>  and other Type -> Coder maps in the same file should prevent insertion if 
> the type already exists and the coders aren't equal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (ae1f547 -> a16abdf)

2018-01-23 Thread tgroh

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ae1f547  Merge pull request #4272
 add b24800d  Register Environments in SdkComponents
 new a16abdf  Merge pull request #4464

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/CombineTranslation.java  | 12 +-
 ...dTranslatorRegistrar.java => Environments.java} | 20 ++---
 .../core/construction/ParDoTranslation.java| 15 -
 .../runners/core/construction/ReadTranslation.java | 26 +-
 .../core/construction/RehydratedComponents.java|  8 +++
 .../runners/core/construction/SdkComponents.java   | 22 +-
 .../construction/WindowingStrategyTranslation.java |  2 ++
 .../core/construction/ReadTranslationTest.java |  4 ++--
 .../construction/RehydratedComponentsTest.java | 14 
 .../dataflow/PrimitiveParDoSingleFactory.java  |  7 ++
 10 files changed, 102 insertions(+), 28 deletions(-)
 copy 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{TransformPayloadTranslatorRegistrar.java
 => Environments.java} (58%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4464

2018-01-23 Thread tgroh

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a16abdfbb489f2f3aba3cf5f54e8205cb728a9ff
Merge: ae1f547 b24800d
Author: Thomas Groh 
AuthorDate: Tue Jan 23 12:45:10 2018 -0800

Merge pull request #4464

Register Environments in SdkComponents

 .../core/construction/CombineTranslation.java  | 12 
 .../runners/core/construction/Environments.java| 35 ++
 .../core/construction/ParDoTranslation.java| 15 +-
 .../runners/core/construction/ReadTranslation.java | 26 +---
 .../core/construction/RehydratedComponents.java|  8 +
 .../runners/core/construction/SdkComponents.java   | 22 +-
 .../construction/WindowingStrategyTranslation.java |  2 ++
 .../core/construction/ReadTranslationTest.java |  4 +--
 .../construction/RehydratedComponentsTest.java | 14 +
 .../dataflow/PrimitiveParDoSingleFactory.java  |  7 +
 10 files changed, 125 insertions(+), 20 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3970

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Updated] (BEAM-490) Swap to using CoGBK as grouping primitive instead of GBK

2018-01-23 Thread Henning Rohde (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-490:
---
Labels: backwards-incompatible portability  (was: backwards-incompatible)

> Swap to using CoGBK as grouping primitive instead of GBK
> 
>
> Key: BEAM-490
> URL: https://issues.apache.org/jira/browse/BEAM-490
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Luke Cwik
>Priority: Major
>  Labels: backwards-incompatible, portability
>
> The intent is for the semantics of both GBK and CoGBK to be
> unchanged, just swapping their status as primitives.
> CoGBK is a more powerful operator then GBK allowing for two key benefits:
> 1) SDKs are simplified: transforming a CoGBK into a GBK is trivial while the 
> reverse is not.
> 2) It will be easier for runners to provide more efficient implementations of 
> CoGBK as they will be responsible for the logic which takes their own 
> internal grouping implementation and maps it onto a CoGBK.
> This requires the following modifications to the Beam code base:
> 1) Make GBK a composite transform in terms of CoGBK.
> 2) Move the CoGBK from contrib to runners-core as an adapter*. Runners that 
> more naturally support GBK can just use this and everything executes exactly 
> as before.
> *just like GroupByKeyViaGroupByKeyOnly and UnboundedReadFromBoundedSource



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (7a4e98f -> ae1f547)

2018-01-23 Thread tgroh

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7a4e98f  Merge pull request #4455 from aaltay/comb
 add 10a6e92  Added an example pipeline that uses stateful processing to 
output team score every time it passes a new multiple of a threshold.
 new ae1f547  Merge pull request #4272

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/examples/complete/game/GameStats.java |  22 +-
 .../examples/complete/game/HourlyTeamScore.java|   6 +-
 .../beam/examples/complete/game/LeaderBoard.java   |  19 +-
 .../examples/complete/game/StatefulTeamScore.java  | 228 +
 .../examples/complete/game/injector/Injector.java  |  15 +-
 .../complete/game/utils/GameConstants.java |  23 ++-
 .../complete/game/StatefulTeamScoreTest.java   | 208 +++
 7 files changed, 467 insertions(+), 54 deletions(-)
 create mode 100644 
examples/java8/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java
 copy 
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectKinds.java
 => 
examples/java8/src/main/java/org/apache/beam/examples/complete/game/utils/GameConstants.java
 (60%)
 create mode 100644 
examples/java8/src/test/java/org/apache/beam/examples/complete/game/StatefulTeamScoreTest.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4272

2018-01-23 Thread tgroh

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ae1f54765fe109a75ecaee58071de135e0504b09
Merge: 7a4e98f 10a6e92
Author: Thomas Groh 
AuthorDate: Tue Jan 23 11:40:05 2018 -0800

Merge pull request #4272

[BEAM-3362] Add a java example that uses stateful processing.

 .../beam/examples/complete/game/GameStats.java |  22 +-
 .../examples/complete/game/HourlyTeamScore.java|   6 +-
 .../beam/examples/complete/game/LeaderBoard.java   |  19 +-
 .../examples/complete/game/StatefulTeamScore.java  | 228 +
 .../examples/complete/game/injector/Injector.java  |  15 +-
 .../complete/game/utils/GameConstants.java |  35 
 .../complete/game/StatefulTeamScoreTest.java   | 208 +++
 7 files changed, 489 insertions(+), 44 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3969

2018-01-23 Thread Apache Jenkins Server

See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3968

2018-01-23 Thread Apache Jenkins Server

See

[beam] 01/01: Merge pull request #4455 from aaltay/comb

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7a4e98f2e128ca537f8546b27cdf12763640e863
Merge: 94d5c0b a69f6b6
Author: Ahmet Altay 
AuthorDate: Tue Jan 23 10:26:29 2018 -0800

Merge pull request #4455 from aaltay/comb

[BEAM-2937] Disable combiner lifting optimization in python DataflowRunner 
for fnapi

 sdks/python/apache_beam/examples/wordcount_fnapi.py | 8 +---
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 9 +
 2 files changed, 10 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (94d5c0b -> 7a4e98f)

2018-01-23 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 94d5c0b  Merge pull request #4437 from robertwb/runner-api-combine-fn
 add 3be0467  Disable combiner lifting optimization in DataflowRunner for 
fnapi
 add a69f6b6  Disable combiner lifting when only the streaming flag is set.
 new 7a4e98f  Merge pull request #4455 from aaltay/comb

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/examples/wordcount_fnapi.py | 8 +---
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 9 +
 2 files changed, 10 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #826

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1023015293746055132.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8502518211590891455.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4638467709473828950.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6761367883247122626.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3637916690236816226.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #50

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1227952887991880109.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3412667410133818628.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6392236304748575308.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3007537559499069631.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6498917499064066983.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1278401681578259127.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #58

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3605254597084213766.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2510924964049932985.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins144855623945818342.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3604036589648694420.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins83309261075490444.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1020898477529349549.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #50

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3890523572967121927.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5120524593888310806.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5069485417035276238.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7473184852000322217.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins408899140858348633.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3680717153504015148.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #51

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3651070016045587915.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3206609034669921079.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5892398163831504205.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4437760236197324100.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6264032114795023926.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8115078931036602527.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

[jira] [Assigned] (BEAM-3522) Review and update the references of maven to gradle in the source code

2018-01-23 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3522:
--

Assignee: (was: Davor Bonaci)

> Review and update the references of maven to gradle in the source code
> --
>
> Key: BEAM-3522
> URL: https://issues.apache.org/jira/browse/BEAM-3522
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Priority: Major
>
> Multiple files in the source code containe references in their documentation 
> to use of maven commands, this should be updated into the corresponding 
> gradle ones or be moved into the website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3522) Review and update the references of maven to gradle in the source code

2018-01-23 Thread JIRA

Ismaël Mejía created BEAM-3522:
--

 Summary: Review and update the references of maven to gradle in 
the source code
 Key: BEAM-3522
 URL: https://issues.apache.org/jira/browse/BEAM-3522
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Ismaël Mejía
Assignee: Davor Bonaci


Multiple files in the source code containe references in their documentation to 
use of maven commands, this should be updated into the corresponding gradle 
ones or be moved into the website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3521) Review and update the references of maven to gradle in the website

2018-01-23 Thread JIRA

Ismaël Mejía created BEAM-3521:
--

 Summary: Review and update the references of maven to gradle in 
the website
 Key: BEAM-3521
 URL: https://issues.apache.org/jira/browse/BEAM-3521
 Project: Beam
  Issue Type: Sub-task
  Components: build-system, website
Reporter: Ismaël Mejía


I suppose that the only maven reference that will stay probably should be the 
one on the maven-archetype but I suppose this will be decided later on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3520) Update testing documentation with gradle info

2018-01-23 Thread JIRA

Ismaël Mejía created BEAM-3520:
--

 Summary: Update testing documentation with gradle info
 Key: BEAM-3520
 URL: https://issues.apache.org/jira/browse/BEAM-3520
 Project: Beam
  Issue Type: Sub-task
  Components: build-system, testing, website
Reporter: Ismaël Mejía






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3396) Update docker development images to use the Gradle build

2018-01-23 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336099#comment-16336099
 ] 

Ismaël Mejía commented on BEAM-3396:


Since Beam includes the wrapped version of gradle with the build it will only 
be needed to inherit from the jdk image instead of the maven one and it will be 
ok.

> Update docker development images to use the Gradle build
> 
>
> Key: BEAM-3396
> URL: https://issues.apache.org/jira/browse/BEAM-3396
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Ismaël Mejía
>Priority: Minor
>
> The docker development images introduced recently are part of the ongoing 
> work on getting reproducible builds on Beam and they should be updated as 
> part of the move to gradle.
> https://beam.apache.org/contribute/docker-images/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4776

2018-01-23 Thread Apache Jenkins Server

See

[jira] [Assigned] (BEAM-3432) Merge sdks/java/io/hadoop/jdk1.8-tests into sdks/java/io/hadoop/input-format

2018-01-23 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3432:
--

Assignee: Ismaël Mejía  (was: Jean-Baptiste Onofré)

> Merge sdks/java/io/hadoop/jdk1.8-tests into sdks/java/io/hadoop/input-format
> 
>
> Key: BEAM-3432
> URL: https://issues.apache.org/jira/browse/BEAM-3432
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Ismaël Mejía
>Priority: Major
>
> We could potentially get rid of the inner sdks/java/io/hadoop/input-format 
> project and create one sdks/java/io/hadoop project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3484) HadoopInputFormatIO reads big datasets invalid

2018-01-23 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335757#comment-16335757
 ] 

Ismaël Mejía commented on BEAM-3484:


Sure I will take a look, but first I will tackle BEAM-3432.

> HadoopInputFormatIO reads big datasets invalid
> --
>
> Key: BEAM-3484
> URL: https://issues.apache.org/jira/browse/BEAM-3484
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Łukasz Gajowy
>Assignee: Ismaël Mejía
>Priority: Major
> Attachments: result_sorted100, result_sorted60
>
>
> For big datasets HadoopInputFormat sometimes skips/duplicates elements from 
> database in resulting PCollection. This gives incorrect read result.
> Occurred to me while developing HadoopInputFormatIOIT and running it on 
> dataflow. For datasets smaller or equal to 600 000 database rows I wasn't 
> able to reproduce the issue. Bug appeared only for bigger sets, eg. 700 000, 
> 1 000 000. 
> Attachments:
>   - text file with sorted HadoopInputFormat.read() result saved using 
> TextIO.write().to().withoutSharding(). If you look carefully you'll notice 
> duplicates or missing values that should not happen
>  - same text file for 600 000 records not having any duplicates and missing 
> elements
>  - link to a PR with HadoopInputFormatIO integration test that allows to 
> reproduce this issue. At the moment of writing, this code is not merged yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3967

2018-01-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Python #825

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5693644691355979568.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1345450801246266482.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3269586146178397808.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1422334400827537723.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5841384343184748154.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line

Jenkins build is back to normal : beam_PerformanceTests_Spark #1269

2018-01-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #49

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6195263603470345417.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1347871962574970201.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1144091584673175034.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1507649138831093631.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7229631838710505736.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3915172461193034809.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #50

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1317523039871231007.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1960687169593151730.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4503546757445783864.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2478653250304027278.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3615797243721700817.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5187167439715122644.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #57

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7435253641503050367.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6273100848646200429.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2994915599865881128.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5416366775879997982.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5289281681414490321.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins9215782257842032066.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #49

2018-01-23 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0
Commit message: "Merge pull request #4437 from robertwb/runner-api-combine-fn"
 > git rev-list 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7472739475582311035.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4015575398651834530.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5920834654169728282.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5357144032295454875.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3972589624956281986.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3718061100733545755.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from

[jira] [Commented] (BEAM-3492) Spark Integration Tests fail with a Closed Connection

2018-01-23 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335575#comment-16335575
 ] 

Ismaël Mejía commented on BEAM-3492:


[~mariagh] The reason of the error is that 
beam-sdks-java-io-google-cloud-platform is leaking some dependencies into the 
classpath in this particular case it is netty that conflicts with the version 
provided by spark, so I 'forced' the Spark runner execution of the examples to 
use the older version but this can have uninttended consequences.

I filled this JIRA for a proper fix from the Google's IO side BEAM-3519


And about the fact that clicking on the test changed it to 'passed' I also saw 
that one, and I believe is Jenkins magic :), I seriously don't have any idea of 
what is going on with that.

 

> Spark Integration Tests fail with a Closed Connection
> -
>
> Key: BEAM-3492
> URL: https://issues.apache.org/jira/browse/BEAM-3492
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Thomas Groh
>Assignee: Ismaël Mejía
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Example: 
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/16832]
>  
> 2018-01-17T23:52:25.668 [ERROR] 
> testE2EWordCount(org.apache.beam.examples.WordCountIT)  Time elapsed: 14.329 
> s  <<< ERROR!
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: 
> Connection from /127.0.0.1:45363 closed
>   at 
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:68)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform

2018-01-23 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3519:
---
Summary: Shade netty and protobuf in 
beam-sdks-java-io-google-cloud-platform  (was: Shade netty and protobuf in 
google-cloud-platform)

> Shade netty and protobuf in beam-sdks-java-io-google-cloud-platform
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Reuven Lax
>Priority: Major
>
> Google Cloud Platform IOs module leaks some of the classical core 
> dependencies (netty/protobuf) and this can cause conflicts in particular with 
> execution systems that use conflicting versions of such modules. 
> For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3519) Shade netty and protobuf in google-cloud-platform

2018-01-23 Thread JIRA

Ismaël Mejía created BEAM-3519:
--

 Summary: Shade netty and protobuf in google-cloud-platform
 Key: BEAM-3519
 URL: https://issues.apache.org/jira/browse/BEAM-3519
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Ismaël Mejía
Assignee: Reuven Lax


Google Cloud Platform IOs module leaks some of the classical core dependencies 
(netty/protobuf) and this can cause conflicts in particular with execution 
systems that use conflicting versions of such modules. 
For the case there is a dependency conflict with the Spark Runner version of 
netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3506) JdbcIO: Support writing iterables (i.e. collections) of rows instead of only single rows

2018-01-23 Thread Knut Olav Loite (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335566#comment-16335566
 ] 

Knut Olav Loite commented on BEAM-3506:
---

In this specific case: I was testing the Jdbc driver with Beam to check that it 
actually worked.

In general: We try to keep the database as loosely coupled with our 
applications as possible. This way, it's easier to switch to another database 
system if that's needed, or to run the application simultaneously on different 
database systems. Using a proprietary API for database access would make this 
impossible.

> JdbcIO: Support writing iterables (i.e. collections) of rows instead of only 
> single rows
> 
>
> Key: BEAM-3506
> URL: https://issues.apache.org/jira/browse/BEAM-3506
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Knut Olav Loite
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: JdbcIO, jdbc
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> The current JdbcIO write interface expects a PCollection where T is the 
> row to be written. Each instance of T is then added to a batch and written to 
> the database. The user has little control over how many rows will be added to 
> one batch. If JdbcIO would also support writing a PCollection 
> the user would have more control over the number of rows in one batch. 
> Especially when writing to cloud databases, such as Google Cloud Spanner, the 
> batching of multiple rows together is important for performance.
> I already have a solution locally and I will submit a pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

1 2 >

1 - 100 of 103 matches

Mail list logo